DCX:UMENT RESUME 



ED 309 273 



CE 052 895 



AUTHOR 
TITLE 

INSTITUTION 
SPONS AGENCY 

PUB DATE 
GRANT 
NOTE 
PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Safarik, Lynn 

A Qualitative Follow-Up Study of a Training Program 

in Curriculum-Based Vocational Assessment. 

California State Univ., Long Beach. 

Office of Special Education and RehabiJ native 

Services (ED), Washington, DC. 

Aug 89 

G008630057 

203p. 

Reports - Research/Technical (143) 
MF01/PC09 Plus Postage. 

*Disal3ilities; Followup Studies; Instructional 
Effectiveness; *Outcomes of Education; Program 
Effectiveness; Secondary Education; Special 
Education; *Teacher Education; *Teacher Improvement; 
Vocational Education; *Vocational Evaluation 



ABSTRACT 

A qualitative follow-up study was conducted to 
determine the impacts of a program to train teachers in the use of 
curriculum-fcased vocational assessment (CBVA) for students with 
handicaps. The study examined the extend to which CBVA training 
content was used in the organization and delivery of assessment 
services to students with handicaps at the training sites. Four 
inservice and six preservice sites were included in the study; 28 
vocational and special education personnel were interviewed and 
observed during onsite visits and by telephone. An analysis of within 
and across-site data revealed a pattern of increasingly comprehensive 
outcome levels at the sites. Findings also included the 
identification of key implementation variables. Three site 
features — the role of the system advocate, the role of the external 
supporters, and the nature of the work environment — appeared to be 
directly related to the level of training outcomes. A fourth, 
continued training, was found to affect implementation most markedly. 
A set of guidelines for training and implementation was also 
identified. Major recommendations included an incremental approach to 
training and use of a site assessment for the selection of training 
sites and implementation strategies. (Interview schedules, summary 
sheets, cind 27 references are appended.) (YLB) 
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ABSTRACT 



A QUALITATIVE FOLLOW-UP STUDY OF A TRAINING PROGRAM 
IN CURRICULUM-BASED VOCATIONAL ASSESSMENT 

By 

Lynn Safarik 
August 1989 

This qualitative follow-up study was conducted to 
determine the impacts of a field-test training program in 
curriculum-based vocational assessment for students with 
handicaps* Four in-service and six preservice sites were 
included in the study; a total of twenty-eight vocational 
and special education personnel were interviewed and 
observed during on-site visitations and by telephone* The 
study examined the extent to which CBVA training content 
was used in the organization and delivery of assessment 
services to students with handicaps at the training sites. 

An analysis of with-in- and across-site data revealed 
a Pattern of increasingly comprehensive outcome levels at 
the sites. Findings also included the identification of 
key implementation variables and a set of guidelines for 
training and imp.lementation. Major recommendations 
included an incremental approach to training and the use of 
a site assessment for the selection of training sites and 
implementation strategies. 
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Chapter 1 
Introduction and the Problem 



Introduction 

Innovation and implementation are the basic elements 
of change in schools. If teachers are to incorporate the 
growing body of research on effective teaching into their 
professional behavior r they must receive on-going training. 
In-service and preservice training programs are the means 
by which new concepts, ideologies, and procedures are 
learned and integrated into a teacher's repertoire of 
classroom practices. Evaluation of these teacher training 
programs is necessary to provide information about effects 
and outcomes of newly acquired professional knowledge. The 
cycle is complete when evaluation data are used for program 
improvement and future planning. 

Impact evaluation is a process by which the outcomes 
and effects of a training program are examined following a 
trainee's exit from the program and entry into the 
employment setting (Albright and Markel 1982). This 
follow-up activity focuses on changes in the trainee's 
performance and changes within the work place. Results of 
the impact evaluation can be used for program planning and 
improvement, to form a base for local decision making, or 

1 
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to meet external requirements such as a federal funding 
source (Brinkerhof f and others 1983; Neilsen and Turner 
1985) • 

The Purpose of the Study 

The purpose of this study was to conduct an impact 
evaluation of a personnel preparation program designed to 
train special and vocational education personnel in the use 
of curriculum-based vocational assessment practices to 
assist handicapped students in secondary vocational 
education programs. The training program represented the 
field-test phase of a three-year national project in which 
a series of training modules in curriculum-based assessment 
and planning strategies was developed, tested, and refined 
(Albright and Cobb 1985)* 

The CBVA training program was primarily developed in 
response to the Carl Perkins Vocational Education Act of 
1984, which recognized a need for a full range of 
assessment and planning services for handicapped students 
which directly relates to their vocational curricula^ The 
CBVA training program tested a systematized approach to 
providing these assessment services • 

The first and second project years (1986-88) included 
preservice training at six universities and in-service 
training at three secondary schools at various locations 
throughout the country* The in-service training groups 



n 
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consisted of vocational instructors, special education 
teachers, guidance counselors, administrators, and 
vocational special education resource personnel. Pre- 
service trainees were professionals from a wide variety of 
backgrounds including rehabilitation services, special 
education, and vocational education. The training program 
entailed instruction of curriculum-based assessment 
principles and practices through the use of a series of 
modules and practical assignments. Formative evaluation 
data were collected from trainees and trainers during and 
at the conclusion of the training process. During the 
third project year (1988-89), the modules were refined and 
an accompanying trainer's manual was developed. These 
products were also disseminated at a national convention 
and a series of regional workshops during the third year. 

The present study examined the extent to which CBVA 
training content was used in the organization and delivery 
of assessment services to handicapped students at the 
training sites. Specifically, the impact evaluation 
follow-up study addressed three objectives as identified by 
Albright and Cobb (1988b): 

(1) to determine the extent to which former 
trainees have implemented curriculum-based vocational 
assessment procedures ii> their programs; 

(2) to examine the impact of these procedures on 
system policy and practices; and, 

(3) to assess trainee satisfaction with the 
preparation received through the field-test program. 
(p-21) 
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This study consisted of a series of on-site interviews 
and observations to determine impacts of training at the 
thiee in-service sites and a sample of the six preservice 
training sites. The study explored changes in 
participants' knowledge, attitudes, and feelings about CBVA 
practices, and changes wiLhin the work environment as a 
result of the training process. By questioning 
participants and observing them in the work setting, a 
richer contextual understanding of impacts were obtained. 

Since this study is part of the larger national 
project, the results will be used by project staff in 
reporting the total project to the funding agency, the 
Office of Special Education and Rehabilitative Services, 
US Department of Education. The results of the study 
will also be used by project personn.^l in refining the CBVA 
training program. 

The Importance of the "^tudy 

Vocational education legislatit;.., r^rticularly the 
Carl D. Perkins Vocational Education Act of 1984, reflects 
a primary mission of vocational education to assure equal 
opportunity and access to special needs populations 
(Phelps 1986). With this trend likely to continue with 
the reauthorization of the Perkins Act in 1989 (Sathre 
1987) , the development of programs which respond to federal 
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guidelines for providing services for special needs 
students will remain as a priority in vocational education. 

Specifically, sections 204 (C) (1) and (2) of the 
Perkins Act support the use of the curriculum-based 
assessment conceptual model in training vocational and 
special education personnel. The CBVA orientation utilizes 
assessment practices that are directly linked to the 
student's curriculum. Unlike standardized tests, 
curriculum-based assessment primarily occurs in the 
classroom, is administered by teachers and support 
personnel who are directly involved with the student's 
daily instruction, and uses informal and direct 
instrumentation. Curriculum-based assessment information 
can then be used to enhance instructional decisions related 
to the student's achievement and aid in determining the 
need for support services. These decisions are based on 
the environmental contingencies of a specific classroom and 
vocational program. 

The extensive service provisions in the Perkins Act 
include guidelines for program information dissemination 
and transitional services assessment. Present assessment 
models lack practical guidelines in these areas (Albright 
and Cobb, 1987). The CBVA approach, which is based on a 
concept of continuous assessment, begins with the 
dissemination of program information to a student and his 
or h^r parents prior to entry into vocational education and 
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ends with planning and support for the student after 
completion of the vocational program and as he or she 
enters an employment situation and/or further training* 

Although the concept of curriculum-based assessment is 
not newr examples of the systematic training procedures as 
developed in this project are virtually absent in the 
literature (Albright and Cobb 1987) • The CBVA project 
established a training program in which vocational 
education and special education service personnel learn to 
collaborate in the use of CBVA assessment and planning 
techniques to make decisions about support services for 
special needs students. As suggested earlier, the project 
involved extensive field-testing and validation of the 
modules from a variety of professionals. 

The proposed study, which is the impact evaluation of 
the CBVA training program, will yield information which can 
be used to improve the training process and products 
outlined in the larger project. If training efforts to 
improve curriculum-based assessment and planning processes 
for handicapped students in secondary vocational education 
programs are to continue, impact evaluations are necessary 
to determine the efficacy of these efforts. 
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Li?nitations of the Study 

The study was limited in that detailed information 
about the participants of uhe follow-up study prior to CBVA 
training was not systematically collected* Ideally, 
information regarding assessment prac^. ices at each site and 
specific job functions of trainees prior to the field-test 
training would have established a basis for comparison. 
However, due to the demands of mounting the field-test 
program in multiple sites, the collection of these data at 
the outset of training was not reasonable at that time. 

Furthermore, the fact that training process and 
products were being f ield-te.^ted at the sites, limits 
expectations about training impacts on participants and 
their work r.iaces. The distinction between "field-test" 
and "actual" training is addressed in Chapter 3, along with 
an explanation of the implications of this limitation for 
the follow-up design. 

Definition of Terms Used 

The following terms are defined in order to clarify 
their meaning within the context of this study. 

Cross-sectional study . A study in which data are 
collected at one point in time from several categories of 
individuals included in a larger population (Asche and 
O'Reilly 1979) 
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Curriculum-baged vocational assessment s A continuous 
process used to answer questions about the instruction and 
special service needs of individual students as they enter 
into and progress through specific vocational education 
programs (Albright and Cobb 1988a). 

Evaluation . The systematic investigation of various 
aspects of professional development and training programs 
to assess their merit or worth (Brinkerhoff and others 
1983). 

Expert judges sampling . A variation of purposive 
sampling by which a population is selected from those 
persons with exceptional expertise about certain conditions 
or factors of interest (Brinkerhoff and others 1983). 

Extreme group sampling . A variation of purposive 
sampling by which a population is selected by seeking out 
conflicting or extreme viewpoints (Brinkerhoff and others 
1983). 

Field-test . The process of applying a principle, 
process, or skill in a realistic setting to collect 
information for purposes of refinement, improvement, and 
future planning (Brinkerhoff and others 1983). 
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Follov-up studv > The attempt to secure educational r 
employment, or other information from or about former 
students for purposes of planning, reporting, evaluation, 
or conducting research (Asche and O'Reilly 1S79). 

Formative evaluation ^ Evaluation used for the 
improvement and development of an ongoing activity, or 
program or product (Brinkerhoff and others 1983). 

Grapevine sampling ^ A variation of purposive sampling 
by which a population is selected on the basis of explicit 
directions from a prior member of the sample (Brinkerhoff 
and others 1983). 

Impact evaluation . Evaluation process which examines 
the influence of the program on its graduates and their 
respective work environments (Albright and Cobb 1988b). 

In-service training . Professional development 
activities which occur simultaneously as an individual 
performs while employed in his/her professional position. 

Key informant sampling . The process of selecting a 
population by accessing those persons with the most 
information about certain conditions or situations 
(Brinkerhoff and others 1983). 

Longitudinal study . A study in which data are 
collected at different points in time in order to study 
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changes or explore time-ordered associations (Asche and 
O'Reilly mS) • 

Matrix sampling ^ The process of selecting a sample of 
respondents from a defined population and items from an 
instrument (Brinkerhoff and others 1983). 

One-shot study . A type of follow-up in which a single 

collection of data is planned. This type of follow-up is 

especially effective for evaluating a new program or 
procedure (Asche and O'Reilly 1979). 

Preservice training . Professional development 
activities which occur as preparation to an individual's 
employment in the field. 

Purposive sampling . The process of selecting a 
population by accessing those persons with the most 
information about particular conditions or situations. 
Examples of purposive sampling include expert judges 
sampling, key informant sampling, and grapevine sampling 
(Brinkerhoff and others 1983). 

Quota sampling . The process of selecting a population 
sample from certain population categories in proportion to 
the relative size of the category (Brinkerhoff and others 
1983). 
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Recurring study , A type of follow-up system in which 
the same basic information is collected from a different 
group of individuals at regular intervals. It is most 
commonly used to supply information necessary on mandated 
reporting by state and federal agencies (Asche and O'Reilly 
1979) . 

Regional Occupational Program (ROP) , A vocational 
education delivery system that serves high school students 
and adults regardless of the geographic location of their 
residence or high school district, and commonly uses 
instructional methodology such as the community classroom 
and cooperative education to provide students with current, 
relevant vocational instruction. 

Sampling . The process of selecting a proportion of a 
population which is representative of that population. 

Straight random sampling . The process of selecting, 

individuals of a population in such a way that each has an 

equal chance of being selected and each choice is 

independent of any other choice (Best and Kahn 1986). 

Stratification . The process of subdividing a 
population into small homogeneous groups to get a more 
accurate representation (Best and Kahn 1986). 
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Systematic sampling . The process of selecting a 
population by selecting the nth term from a finite list 
(Best and Kahn 1986). 
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Chapter 2 
Review of Related Literature 



A review of the literature on impact evaluation 
processes and methodology initially included a broad survey 
of exemplary follow-up studies in vocational education • 
While follow-up studies of vocational program trainess are 
largely concerned with placement, the relative abundance of 
vocational education follow-ups (Asche and O'Reilly 1979) 
provided numerous examples from which a generali:5able 
pattern of procedural steps could be identified, in an 
effort to focus the literary review on the methodology used 
in impact evaluations of teacher training programs , an 
emphasis was placed on field work research methods, 
including case studies and theoretical papers on 
qualitative and quantitative methods of data collection and 
analysis* 

A review of vocational student and employee follow-up 
studies utilized findings of the comprehensive research 
efforts of Asche and Reilly (1979), Richardson and 
Nieminen (1985), and Wentling (1980). These authors 
provided an overview of the procedures used in follow-up 
studies of students completing vocational education 
programs. A series of similar steps was delineated in each 



13 



ERLC 



Z2 



14 

study, with minor differences in the level of detail at one 
stage or another • 

in Program Evaluation . Brinkerhoff and others (1983) 
outlined procedures for conducting a follow-up evaluation 
of teacher education programs; these procedures were 
consistent with those identified in the vocational 
education program student follow-ups. The following review 
of related literature is organized around a collective 
presentation of these procedural steps, with the exception 
of Reporting the Data, which has been omitted. Material 
relating specifically to impact evaluations of teacher 
training programs was systematically targeted and 
emphasized in the task descriptions. 

1. Focus the Evaluation/Determine 
Goals of the Study 

The establishment of overall goals and primary 

objectives is the foundation from which the follow-up 

design is to emerge. As Jick (1979) indicated, "If the 

research is not clearly focused theoretically or 

conceptually, all the methods in the world will not produce 

a satisfactory outcome" (p. 146). Three broad follow-up 

goals are accountability, program planning, and program 

improvement. Asche and O'Reilly (1979) emphasized the 

importance of identifying potential users of the data as a 

means of identifying system goals. Brinkerhoff and others 
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(1983) reiterated the importance of audience consideration 

in evaluation design, noting that: 

It is important to identify these 'stakeholders' in 
the evaluation as they will provide the basis for some 
of the evaluation questions generated later. 
Evaluators cannot meet relevant needs unless they know 
who will use the evaluation and how, (p. 20) 

The authors stated their perception of program evaluation 
as functional: 

Evaluations should serve the development, 
implementation, and recycling needs of training 
programs. It should be used for one or more purposes; 
to iniprove a particular program (formative); for 
accountability or selection (suramative) ; to motivate, 
increase and gain support of staff and others 
(psychological), (p, xx) 

Impact evaluation focuses on effects and outcomes of a 
training program on the trainees and their respective work 
environments. Albright and Cobb (1988) described the 
impact evaluation as the last of three phases in a 
formative luation, following Phase I, Planning 
Evaluation; and Phase II, Process Evaluation (p. 5). The 
authors cited two primary objectives of the impact 
evaluation of a training program: 

(1.) To determine the extent to which skills taught in 
the training program are being used by the graduate in the 
workplace, and 

(2.) to determine how the application of these skills 
is affecting the work environment (p. 20). 



2» Develop Timelines 
Asche and O'Reilly (1979b) recommended the 
establishment of a general system time frame at the onset 
of the follow-up process. Timing of data collection would 
be considered at this point as well as reporting deadlines. 
Information on time frames gathered early in the process 
would be useful later on, especially when making decisions 
about methodology and instrumentation. 

3. Make Major Operational Decisions 
Based on their research, Asche and O'Reilly (1979b) 
included a premliminary step to the design stage of the 
process. The authors stressed the importance of initially 
making major operational decisions such as how to process 
the data, whether the follow-up should be conducted in- 
house or by a contractor, and whether data should be 
collected continuously or at regualr intervals (p. 38). 
Along these lines Brinkerhoff and others (1983) recommended 
considering the "evaluation setting" prior to the design 
phase. Evaluation setting elements might include organi- 
zational politics, economics, or social patterns. The 
authors offered specific questions regarding these various 
influences and their possible impact on the evaluation 
setting; addressing these questions early on might aid in 
interpreting data later (p. 23). 



4^ Define the Follov-up Population 
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In part/ the task of defining the population will have 
been accomplished by defining the goals and objectives of 
the follow-upo It is important to note that in addition to 
determining relative characteristics of the population, 
this step also provides information about the nature of the 
data to be collected. For example, by determining the size 
of the population and the geographic locations of the 
individual in the population, preliminary decisions about 
whether to use a survey, personal interview, or telephone 
interview can be made (Asche and O'Reilly 1979). 

Ideally, all participants of a training program should 

be included in a follow--up study, when this is not 

-easible, sampling and stratification techniques are 

employed to aid in the selection of a representative group 

(Best and Kahn 1986) . On the subject of sampling, 

Mintzberg (1979) was skeptical of the "significance" of the 

statistical significance of large samples and posed some 

interesting questions: 

What, for example, is wrong with samples of one? Why 
should the researcher have to apologize for them? 
Should Piaget apologize for studying his own children; 
a physicist for splitting only one atom? A doctoral 
student l know was not allowed to observe managers 
because of the "problem" of sample size. He was 
required to measure what managers did through 
quebcionnaires, despite ample evidence in literature 
that managers are poor estimators of their own time 
allocation (e.g. Burns, 1954; Home and Lupton, 1965; 
Harper 1968) . Was it better to have less valid data 
that was statistically significant? (p. 108) 
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5^ Determine Data Needs 
The system goals and objectives should provide a 
framework for determining data needs. A systematic 
determination of data needs is accomplished by developing 
major data categories from program objectives. These data 
categories can be further broken into related data elements 
and finally data items, or questions (Asche and O'Reilly 
1979) . 

Brinkerhoff others (1983) identified these 
evaluation questions as "the building blocks for the 
evaluation" (p. 27). The authors recommended that 
evaluation questions initially be generated from the 
overall purpose of the study, but should be refined, 
revised, and reorganized according to input gathered from 
audiences through conversations, interviews, surveys, and 
group meetings, if time and resources permit. They also 
described six methods for defining evaluation questions 
including (p. 28): 

(1.) Analysis of the Object: Questions are related 
to critical junctures, areas of staff concern, major 
milestones, key objectives, etc.; 

(2.) Use of Theoretical Framework: Questions are 
derived from model's key points; 

(3.) External Expertise and Experience: Questions 
identified by experts in the field or literature review; 
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(4*) Interaction with Key Audiences: Questions 
identified by audience as most important; 

(5*) Definition of the Purpose for Evaluation: 
Questions based on a logical analysis of the purpose for 
the evaluation; and, 

(6.) "Bonus" Questions: Additional questions asked 
for secondary purpose — public relations, research, 
marketing, etc. 

The data needs of the impact evaluation can be 
theoretically organized around a combination of these 
methods, but should be also be based on a preliminary 
design for analyzing the data. 

6. Collect and Store Participant Data 
Mention of a sub-category of data needs is made in 
Richardson and Nieminen's (1985) report, where they refer 
to the collection of identifying information about 
respondents. Addresses, phone numbers, program enrollment 
data, and descriptive data are all items to be included in 
the system records. Although identifying information can 
usually be obtained from records, it may be included in the 
instrument to provide a cross-check. 

7. Develop a Follow-up Methodology 
The methodology of the study refers to the procedures 
and methods employed in actual data collection. The 
function of the methodology is to address the evaluation 

ERIC 25 
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questions which direct the entire study. Areas to consider 
when designing the methodology include: the nature of data 
to collected r the assessment and selection of data 
collection procedures, sampling methods / instrumentation 
development and evaluation/ validity and reliability 
issues/ and finally, efficiency of the design (Brinkerhoff 
and others 1983) . 

Decisions about methodology are based upon the 
evaluation questions posed/ time and financial resources/ 
and anticipated credibility and usefulness in terms of 
analysis/ interpretation/ and reporting (Asche and O'Reilly 
1979; Brinkerhoff and others 1983; Richardson and Nieminen 
1985)* in light of the complex interplay of variables 
involved in designing the methodology/ this section is 
subdivided to accommodate the various factors/ issues/ and 
perspectives presented in the research on impact evaluation 
methodology. 

Study Type 

According to Asche and O'Reilly (1979)/ there are four 
basic types of follow-up studies: one-shot/ recurring/ 
cross-sectional/ and longitudinal. The one-shot is best 
used for collecting data about a specific program and with 
a limited population. Data are collected at one time only 
and study objectives are very specific The limited nature 
of this type of study can reduce costs; however/ costs may 
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be incurred at the planning and design phase in an effort 




to ensure a high quality of data# 




For most state and federally mandated follow-ups, a 




recurring study is used. This type of study allows data to 




be collected from the same population at regular intervals. 




Asche and O'Reilly (1979) noted three advantages to using 




recurring studies: the?' collect data that can be used to 




describe trends and changes; they allow for the 




establishment of a rapport with the participants which will 




enhance a high rate of return; and they involve repetition 




which provides an opportunity for improving the data 




collection instrument with experience in its use (p. 60). 




in a cross-sectional study, an effort is made to con- 




currently collect data from different groups of a 




population. This type of study yields information that may 




be used to describe contrasts or trends. The authors 




(Asche and O'Reilly) cautioned, however: 




The use of cross-sectional data to explore time- 
ordered relationships must be done with caution, 
especially if respondents are asked to recall previous 
incidents. Also, it must be assumed that the 
conditions affecting individuals have remained 
constant for all categories of individuals in the 
follow-up. (p. 61) 




A longitudinal study is the type most used to 




investigate time-ordered associations. A panel 




longitudinal study is one in which the same individuals are 




followed-up at different points in time — perhaps one. 




three, and five years after they have left a program. 
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Descriptions of trends are most accurate when based on data 
gathered through longitudinal studies, since the same group 
is followed over time. However/ because of the cost/ time, 
and effort involved/ longitudinal follow-up studies are not 
common in the fields of vocational education and special 
education (Albright and Markel 1982). 

Sampling 

Another major facet of the follow-up methcdclcgy is 
the determination of the population size. The study goals 
and objectives/ system costS/ and data quality are factors 
to be considered when making this decision. When a 
population is small/ a "census/" of follow-up that attempts 
to collect data from all members of a population/ is 
desirable. Most existing follow-up studies in vocational 
education/ use a census design (Asche and O'Reilly 1979). 

When time and/or money do not allow for a census 
design; sampling is used. Sampling is the selection of a 
portion of a whole 9 oup to represent the whole group. 
(Brinkerhoff and others 1983/ p. 89). The intent is to use 
data generated from the sample to generalize to the larger 
populTition. Brinkerhoff and others (1983) noted that 
sampling can be used in relation to people or things. He 
explained that/ for example/ "If your purpose is to make 
diagnostic decisions about trainees/ you have to evaluate a 
sampling of competencies from each trainee" (p. 89). 
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Two general options in sampling methods include random 

and purposive (Brinkerhoff and others 1983/ p. 89). ;7hGa 

the major concern is that the sample is a true 

representation of the total population, random methods are 

used to eliminate bias. The authors listed four random 

sampling methods: straight random sampling, quota 

sampling; stratified sampling, and matrix sampling (pp. 90- 

91). When a particular perspective or group representation 

is desired, purposive methods are used. Four purposive 

sampling methods were described as well: key informant, 

expert judges, extreme groups, and grapevine sampling 

(pp. 90-91). Brinkerhoff and others (1983) illustrated a 

case for the selection of purposive sampling in a follow-up 

of participants of an in-service training program: 

For example, consider the typical case of sending a 
follow-up survey to graduates of a program or 
participants in an in-service. Usually, a random 
sample of attendees is drawn to receive the survey. 
But, this method assumes that you wish to make an 
inference (an estimate) about all attendees. Very 
often » however, the purpose is to make some judgments 
about the program (curriculum, workshop, etc.) itself, 
not about the typical or average attendee Thus, it 
might make more sense to draw a sample of those whose 
judgments and opinion could mean the most or be most 
useful. This might be high-scoring graduates (or poor 
ones), or specially qualified attendees, persons with 
a lot of experience, etc. (p. 92) 



Data Collection Methods 

Traditionally, data collection methods for follow-up 
studies include the personal interview, telephone 
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interviewjr and the mail survey, or questionnaire (Asche 
and O'Reilly 1979). Other methods include: tests, ratings, 
and frequencies; naturalistic observations; ethnographic 
descriptions; case studies; and literature review 
(Brinkerhoff and others 1983) • Data collection procedures 
were also categorized by Brinkerhoff and others (1983) as 
being either quantitative or qualitative in nature: 

Quantitative — Using these procedures results in 
numerical data. We call such data "convergent" in 
that phenomena (opinions, performance, behaviors) are 
"reduce" and put into categoriees that can be 
assigned a number. Then these numbers can be 
summarized and otherwise manipulated. 

Qualitative — These procedures produce narrative 
information. (While narrative information can be 
converted into numerical categories, that would 
usually serve an antiethical purpose.) Qualitative 
procedures tend to capture broader and more open-ended 
perspectives about complex phenomena. (pp. 84-85) 

Most often, a variety of procedures is used to answer 

questions in a follow-up study, and in fact, the literature 

suggested that a complementary mix of quantitative and 

qualitative methods was useful in collecting productive 

data in the field (Miles 1979, Van Maanen 1979). On the 

benefits of multiple data collection methods, Brinkerhoff 

and others (1983) stated: 



this is especially true when whatever you are 
measuring (e.g., program, in-services, or trainees) is 
complex and made up of a number of variables, as one 
measure rarely captures the richness and variety of 
such evaluation "objects." (p. 135) 

King and others (1987) recommended that prior to organizing 

details of an evaluation plan, the evaluator should decide 
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on which overall orientation is most appropriate in 
providing information that will be most useful to the 
designated audience and most feasible in terms of human and 
material resources. The approach may be qualitative, 
quantitative, or a combination of the two* 

Quantitative Procedures 

Traditionally, educational program evaluation has 
utilized the scientific methods borrowed from biology and 
physics (King and others 1987) • The authors explained that 
quantitative methods are philosphically based on the view 
that, "through repeated studies, science comes even closer 
to the truth; describing with increasing accuracy an 
objective reality that exists apart from any individual** 
(p. 21). Using such an approach, the evaluator then 
proceeds to investigate the questions typically posed by a 
set of hypotheses. 

The mail survey or questionnaire is the most widely 
used form of follow-up data gathering devices in vocational 
education (Richardson and Nieminen 1985). The authors 
maintained that these self-report measures have the 
potential for providing the most reliable data, are 
convenient, save time and money, and require fewer 
personnel than the personal interview. 

King and others (1987) listed other advantages to 
questionnaires: they can be answered anonymously; they 
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allow the respondent time to think before responding; they 
can be administered to many people at various sites at the 
same time; they can be mailed; and they impose uniformity 
on the information collected by asking all the respondents 
the same questions (p. 47)* On the other hand, although 
more respondents can be reached regardless of geographic 
location, the rate of return is not as high as with the 
personal interview, "since persuading people to complete 
and return questionnaires is sometimes difficult" (King and 
others 1987, p. 47). Furthermore, questionnaires do not 
provide the flexibility of interviews and the written 
response may be a more difficult mode of expression for 
some people (King and others 1987, p* 47). 

There is an abundance of information in the literature 
on designing an effective questionnaire (Asche and O'Reilly 
1979; King and others 1987; Wentling 1980). All provided 
criteria and guidelines for designing questionnaires that 
are reliable, provide valid data, and have high return 
rates. 

Wentling (1980) noted two formats used in the 
questionnaire, either selected or constructed. A closed- 
reponse or selected questionnaire lends itself to easy 
scoring and reporting; whereas an open-ended or constucted 
questionnaire has the potential for richer feedback, but is 
time-consuming to score and analyze. King and others 
(1987) recommended using a principally closed-response 



format r with at least one open-ended question on every 
questionnaire. They offered this rationale: "Giving 
people the opportunity^ to write down their concerns alerts 
them to the importance of their perspective and provides 
the evalutation helpful information for guiding program 
activities" (p. 77). 

In addition to surveys, knowledge tests are 
quantitative measures that can be used to objectively 
measure participants' knowledge and cognitive skills^ 
Performance tests can be used to measure job-related and 
specific task behaviors. Observations can be quantified 
using behavior observation check-lists and interaction 
analysis. Judgmental ratings, inventory checklists, q- 
sorts, and delphi techniques are other examples of 
quantitative procedures (Brinkerhoff and others 1987). 

Qualitative Procedures 

Qualitative methodology has been described as an 
"emergent" strategy (Van Maanen 1979) with "patterns 
realized despite the absence of intentions" (p. 105) . In 
contrast to traditional quantitative methods, "qualitative 
methods are based on the belief that reality does not exist 
apart from someone's perception of it, and the evaluator at 
best describes the details he or she observes, allowing 
language and events selected by participants to tell their 
own story" (King and others 1987, p. 22). This approach 



borrows techniques used in anthropology and sociology and 

is variously described as "naturalistic," "exploratory," 

"holistic," "ethnographic," and "direct" research* The 

qualitatively-oriented evaluator typically uses interviews, 

on-site observations, and participant-observations, for 

close-up descriptions and is not limited to preconceived 

notions about the study (Light 1979; Mintzberg 1979). 

Proponents of qualitative methodology cited many 

attractive qualities of the approach for evaluative 

studies. Miles (1979) described qualitative data as being 

"rich, full, earthy, holistic, and real" (p. 117), and as 

having a quality of "undeniability. " Other advantages of 

qualitative data were mentioned in the literature, 

including their high face validity, the fact that their 

collection requires minimal front-ond instrumentation, and 

their tendancy to preserve chronological flow. King and 

others (1987) provided other reasons for utilizing a 

qualitaitve approach: 

unhampered by preconceptions and prescriptions, 
qualitative inquirers may set their sights on catching 
the true flavor of a program, discovering the unique 
set of elements that make it work and conveying them 
to the evaluators* audience. 

... A qualitative approach may be necessary if 
there is no written plan for the program you are 
evaluating.. .Even if there is a plan, it may be vague 
or, from your perspective, unrealistic to implement. . 
. . You may discover that the program has varied so 
much from site to site that common features are not 
apparent at first, (p. 22) 
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The interview, an instrument most often used in 
qualitative research, is best suited to studies involving a 
relatively small population or in studies in which the 
attitudes, perceptions, and personal feelings of 
participants are being sought (Wentling 1980). Interviews 
can use flexible data gathering techniques or rely on a 
structured schedule. Interviews may range from informal 
personal conversations with program personnel to highly 
quantitative formats using a closed-response interview 
schedule (Brinkerhoff and others 1983; King and others 
1987, Wentling 1980). 

in a structured interview, the role of the interviewer 
is limited; his or her main function is to read specific 
questions in a precise order and to record responses. In 
an unstructured interview, the interviewer presents a few 
broad areas of concern, and stimulates the respondent to 
expound on these topics. The interviewer is not bound to a 
predetermined format, but instead may adapt a set of 
interview guidelines when necessary to explore areas of 
implementation that were either unplanned or evolved 
differently from the plan (Brinkerhoff and others 1983; 
King, 1987; Wentling 1980). 

An unstructured interview requires considerable 
interviewing skill, since there is no specific sequence in 
a flexible interview, the interview flows much like a 
conversation and the respondent can easily go off on 
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tangents. The skilled interviewer will prepare questions 
designed t^ refocus the interview and to encourage 
responses when a respondent is hesitant* It is possible to 
obtcin accurate and indepth information in an unstructured 
interview since the interviewer can probe the respondent 
for underlying reasons for a response when necessary* 
The literature on interviewing techniques provided guide- 
lines and examples on the use of probes in an unstructured 
interview. Probes were described as short comments made by 
the interviewer to stimulate the respondent to remember 
morer be more specific, supply examples, or to guid^ the 
interview to a more relevant topic. Several examples of 
frequently used probes were cited: "Can you tell me more 
about that?"; "why do you think that happened?"; and, "I 
see. Is there anything else?" (King and others 1987 
p. 81). The authors recommended that probes be inserted 
whenever a respondent makes an emphatic statement in either 
an expected or unexpected direction. The interviewer can 
encourage the respondent to reconsider his/her statement 
simply by rephrasing or repeating the strong response. 
In an unstructured interview, the flexibility of the 
interviewer can allow for unanticipated, but significant 
data. Piore (1979) illustrated this situation as he 
experienced it in his research on civil rights and anti- 
poverty groups: 
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At first/ I develped an elaborate list of questions 
for this "preliminary" part of the interview/ but I 
quickly found that the questions had very little to do 
with the success or failure of the interview. • . • 
The interviewees used my questions as an excuse for 
telling their stories. Since I thought of my initial 
interview as a means of developing a rapport/ this did 
not bother me at first. Indeed/ I was glad to be 
relieved of the burden of keeping the conversation 
going/ and i began to look for ways to get the 
respondent to do his or her thing. Later/ I became 
interested in using the same interview format to 
obtain the answers to a specific list of questions/ 
but I was unable to change the interview process. 
Either I let the repondent tell his or her story/ 
using my questions as an excuse/ or else I forced him 
or her to treat the questions seriously and to give me 
a codable response to each item. If I took the latter 
approach/ the respondents soon lost interest in the 
project and began to concentrate on getting through 
the questionnaire and on to their next appointment. 
In this process/ they often provided misinformation in 
order to avoid an anticipated follow-up question, 
(p. 72) 

Eventually/ the author discovered that the stories of the 
respondents "revealed that the processes of technological 
change and labor allocation/ indeed the basic process of 
business management/ were totally different from the ways 
in which the original prroject had been conceived" (p. 73). 

Van Haanen (1979) also stressed the importance of 
supporting systematic data with anecdotal data/ explaining 
that "while systematic data create the foundation for our 
theories/ it is the anecdotal data that enable us to do the 
building" (p. 113). 

The importance of establishing rapport with the 
respondent was emphasized in the literature. Wentling 
(1980) maintained that a good interviewer must be an 
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understanding^ non judgmental person, must be able to put 
the respondent at ease and get him/her interested in the 
interview and provided some basic principles for 
establishing and maintaining good interviewing relations. 
King and others (1987) explained that efforts to alleviate 
the respondent's initial apprehensions about the interview 
could be made by stating the purpose of the interview, 
exactly what would be required of the respondent and 
stating whether or not the information gathered could be 
kept confidential. An explanation should be made about 
confidentiality and how information from the respondent 
would affect the program. 

Field-testing of the interview can be accomplished 
simply by rehearsing the interview with a colleague, 
spouse, or anyone who could give feedback on how he/she 
interpreted the questions. By using a dry-run, the 
interviewer and interviewee can discuss possible 
misunderstandings, threatening questions, and logical 
inconsistencies (Brinkerhoff and others 1983). Several 
authors recommended advising the tester to be as 
uncooperative as possible to help prepare the interviewer 
for unanticipated answers or hostility (Brinkerhoff and 
others 1983; King and others 1987). 

Interview data may be ai.Jiotaped and transcribed 
later, when this is possible, the tape enables the data 
analyzer to summarize using direct quotes. The problem 
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with taping interviews is that transcription time, in the 
case of dictated write-ups r could take as long or longer 
than the original contact (p. 83). King and others (1987) 
recommended recording data on summary sheets which could 
later be reduced into a narrative report, and added/ that 
when possible/ these summaries should be reviewed by the 
respondent to avoid misunderstandings about responses. 
Miles (1979) noted/ however/ that notes "can't be reduced 
without losing many direct quotes/ earthy data/ etc./ which 
make ethnographic data so useful • . (p. 121). 

The telephone interview shares many of the advantages 
of the personal interview in that it provides personal 
contact and therefore a high rate of response, it is 
commonly used as a secondary data collection method/ 
following a personal interview or a questionnaire (Asche 
and O'Reilly 1979). Like a personal interview/ the 
telephone interview can be costly. The telephone 
interviewer must be trained in interviewing skills as well. 
An interview script may be developed and used to 
standardize the process and increase the reliability of the 
data. Frey (1983) noted the increasing acceptance among 
researchers of the telephone interview/ offering numerous 
factors which have contributed to the trend including the 
high cost and declinir^ response rates of the face-to-face 
interview/ improved telephone technology/ and increased 
accessability to any population. 
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The literature specified various human behaviors 
associated with telephone including the loss of a 

feeling of privacy of telephone interviews versus 
questionnaires (Asche and O'Reilly 1979) and reluctance to 
be totally consumed by a telephone conversation as possible 
explanations for less honesty / more evasiveness, or refusal 
to be interviewed over the telephone (Frey, 1983). 

on-site observation is a data collection method that 

is noted in the literature for the credibility and richness 

of the data it yields (Light 1979; Piore 1979). In 

ethnographic studies, which usually entail at least a year 

devoted to the task, observation is often the primary 

method of data gathering. Sanday (1979) explained, 

"Fieldworkers learn to use themselves as the principals and 

most reliable instruments of observation, selection, and 

interpretation" (p. 20). Because observation places the 

evaluator close to the training setting; he or she is 

able to gain a sense of context and an intuitive feeling 

for a program's implementation. On observation. King and 

others (1987) stated: 

Some evaluators f eeJ ^ in fact, that observation is the 
only method for capturing and aptly describing a 
program's complexity because formal observations take 
a long look at representative parts of the program and 
record accurately what is seen. (p. 85) 

The distinction made by the authors regarding formal 
versus, informal observation techniques is worthy of 
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consideration. While King and others (1987) acknowledged 
both highly structured, quantifiable observations as well 
as unstructured, qualitative observations, they stressed 
the need for a systematic approach to be sure the method is 
rigorous enough to be included as evaluative data. The 
literature provided numerous examples of structured 
observation models including the Stallings and Kaskowitz 
Model Observation System developed at Stanford Research 
Institute (King and others 1987) . 

While quantitative observations involve a 
predetermined plan for what, when, and how observations 
will be made, the qualitative strategy is to allow critical 
issues to emerge through the course of the observations. 
In naturalistic studies, on-going field data are collected 
through a variety of methods in addition to qualitative 
observations, in an effort to enhance confidence in the 
emerging theory. Light (1979) commented on the merits of a 
qualitative approach tn observation in evaluative studies: 
"To find out how a training program really works and to 
get beneath stated objectives and surface behavior, one 
must use observation" (p. 57). Light also expressed his 
view on the limitations of an exclusively qnantified 
approach: 

Observation allows the vital flexibility to 
discover. Scales, instruments, and questionnaires 
require that one presumes prior knowledge of all the 
key forces that shape trainees and all the key 



dimensions on which they change. Worse, they do not 
allow one to measure or discover anything else, 
p. 60) 

Several others described direct observation procedures as 
being a vital part of holistic analysis in which hypotheses 
emerge from the data and are continually tested, retested, 
and altered rather that having been previously set and 
limited by preconceived notions (Jick 1979; Mintzberg 1979; 
Van Maanen 1979). 

Instrumentation is only one component of the 
methodological design. The efficiency of the design 
ultimately depends on the interacting effects of the 
information source, the information collection procedures, 
the evaluation questions, and the particular setting 
(Brinkerhoff and others 1983). while the authors stressed 
flexibility in terms of overall design, tney supplied a 
list of considerations for selecting the data collection 
procedures, which include: (a) availability of data from 
other sources; (b) Training needs of data collectors; (c) 
pilot testing of instruments; (d) interruption potential; 
(e) respondent reactivity to data collection; (f) protocol 
needs; (g) campling bias; (h) reliability; and, 
(i) validity. 

Trianqulation 

The literature on evaluation study methods, included 
various descriptions, case studies, and theoretical papers 
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on direct research methods variously identified as 
convergent methods, multit rait/mult imethod, convergent 
validation, or triangulation (Jick 1979; McClintock and 
others 1979). Evaluation studies were illustrated which 
used combinations of qualitative and quantitative methods 
in a complementary fashion, using convergent data for 
analysis. Jick (1979) presented Denzin's general 
definition of triangulation, "Triangulation is the 
combination of methodologies in the study of the same 
phenomena," but explained that the term originated from 
navigational and military strategies which used multiple 
reference points to located an object's position (pp. 135- 
136). The rationale for such a strategy is that the 
convergence of two or more methods can enhance the validity 
of the data and ensure that the data gathered is a result 
of a behavior or trait and not a method (Jick 1979). 

Jick (1979) made the distinction between two types of 
triangulation: between (or across) methods, and within 
methods triangulation. The across method variation 
involves the use of multiple methods to examine the same 
dimension of a research problem (p. 136). An example of 
"across methods" triangulation might entail interviews, 
observations, and the evaluation of performance records to 
gather data about one research question, in this 
situation, the data collection method varies, but the 
object of the research remains the same. Using across 



method triangulation^ multiple and independent measures can 
be used as a cross validation of data. 

When multiple techniques within a given method are 
used to collect and interpret data/ the method is described 
as "within methods" triangulation* If / for example, an 
evaluator used multiple comparison groups for participant 
observation, or interviews several subgroups to gain 
various perspectives about a particular research question, 
the cross-check enhances reliability of the data (Van 
Maanen 1979) . 

Triangulation designs can range from very simple, 
primitive devices such as scaling, to more sophistocated 
strategies involving complex combinations of qualitative 
and quantitative data collection. Jick (1979) warned of 
misusing triangulation, specifically the parenthetical, 
patronizing use of field observations to strengthen 
statistical results," and of the limitations of such 
misuse, explaining that, "five different variations of the 
same method don't generate five distinct varieties of 
triangulation data" (p. 137) o 

8. Analyze Data 
Once the data have been collected, the process of 
analysis begins. However, major decisions about analysis 
should be made early in the planning stages of the data 
collection design, (Asche and O'Reilly, 1979; Brinkerhoff 
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and others 1983) primarily to determine whether data 
analysis will be quantitative or qualitative in nature* 
The overall orientation of data analysis is closely tied to 
instru-mentation and the purpose of the study. For 
instance, Brinkerhoff and others (1983) noted that 
qualitative analysis is well-suited to deriving meaning 
from richf descriptive materials like interview summaries 
and site visit reports and for special training program 
evaluations. 

The literature reviewed included various assessments 
of the state of research on qualitative data analysis. 
Miles (1979) stated that according to Sieber's research 
"most texts largely ignored the problem of analysis," 
(p. 125) and felt that little was known about making sense 
of large amounts of qualitative data. On the other hand, 
more recent works cited numerous references that provide a 
conceptual framework for conducting qualitative analyses 
and examples of cases studies (Brinkerhoff and others 1983; 
King and others 1987). 

These authors have suggested a process of intertwining 
analysis and data collection, in which questions emerge 
from a preliminary analysis and several cycles of analysis 
follow. Often, the process involves additional data 
collection and re-^assessment of evaluation questions. Both 
sources stressed, however, that a systematic process be 
established at the outset and that each step be documented 



to assure ease of replication* 

The general sequence of analysis procedures was 
presented in various sources as (a) data "clean-up" or 
verification; (b) data organization, or coding; (c) 
identification of conunon themes/ trends, and relationships; 
(d) provisional testing of hypotheses; and, (e) 
interpretation, which is a comparison of descriptive data 
to a referant (Brinkerhoff and others 1983; King and others 
1987; Miles 1979). 

The authors provided similar guidelines for effective 
analysis and warned not to oversimplify or to assume 
unrealistic uniformity in analysis, but to account for 
differential effects and conditions; use multiple 
techniques based on different assumptions; and use direct, 
practical, and affordable methods of analysis* 

Specifically, Brinkerhoff and others (1983) described 
three methods for analyzing narrative data: the social 
phenomena approach, content analysis, and records analysis* 
King and others (1987) presented several methods for 
summarizing open-ended response instruments and narrative 
reports by categorizing. Miles illustrated the use of 
formative evaluation data to facilitate analysis; by using 
site summaries, site updates, and site analyses. His 
analysis on Social Architecture in Education became 
increasingly productive through on-going review, 
validation, and interpretation by project staff and 
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participant feedback. His study also used multi-site 
comparisons to gain perspective on idiosyncratic aspects of 
each site and to correct conceptual bias. 

Van Maanen (1979) presented sieber's list of "rules of 
thumb" for qualitative analysis that were consistant with 
several other sources on the subject of validity and 
reliability of the data. In general, the authors 
recommended using a holistic approach to data 
interpretation, by focusing on the interrelationships of 
multiple and often conflicting evidence, when considering 
the validity of any particular generaa.-.zation, the analyzer 
should look for negative evidence and make predictions such 
as, "if X is true, what else would be true?" Miles 
outlined a similar set of processes for validating 
generalizations also, but posed some interesting questions 
about reliability issues as they relate to qualitative 
analysis: 

the quantitative view of reliability (inter-observer, 
inter-respondent, inter-instrument, or intra- 
respondent over time) is in many respects inapplicable 
in qualitative data research, . . certain kinds of 
reliability must be intentif tally violated in order to 
gain a depth of understanding about the situation 
li.e., the obvserver's behavior must change from 
subject to subject, unique questions must be asked of 
different subjects. ... (p. 126) 

Miles (1979) cited Cuba's work which also stressed that 

"perhaps qualitative research should be seen as 

•auditable', 'conf irmable' , and 'creditable' rather than as 

•reliable' and 'valid' in the usual sense" (p. 127). 
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Chapter 3 
Methods and Procedures 

Introduction 

A sketch of the design for this impact evaluation 
study initially appeared in the proposal of a federally 
funded project conducted through California State 
University, Long Beach from June 1986 to May 1989 (Albright 
and Cobb 1985) . The follow-up study was planned as part 
of the third year activities of this national project. 

During the first project year, a series of personnel 
training modules was developed based on a three-phased 
conceptualization of assessment; answering assessment 
questions about students prior to, during, and after their 
enrollment in a vocational program. The modules were used 
in a variety of in-service and preservice training 
settings during the second project year. In addition to 
training activities, the second project year included the 
collection of formative evaluation data from trainers and 
trainees. During the third project year, prodvict 
publication and dissemination through a national convention 
and several regional workshops occurred. The final phase 
of the project is the present follow-up study, the impact 
evaluation. 
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in the original proposal, the design of the follow-up 
indicated that on-site interviews at the three school 
districts that completed the in-service program would be 
conducted and a representative sample of trainees from the 
six preservice program sites would be interviewed by 
telephone. The general purpose of the follow-up as stated, 
was "to gather data on the impacts of training on 
participants and their work environments" (p. 21). 

While the general features and objectives of the 
follow-up study have essentially remained the same, further 
development of the project, analysis of formative 
evaluation data, and input from project staff members 
resulted in refinements to the preliminary follow-up 
design. 

The following description of methods and procedures 
used in the impact evaluation study is organized according 
to the major procedural steps as they were outlined in the 
previous chapter. Review of the Related Literature. One 
step in particular. Methodology, is subdivided to 
accommodate three major methodological concerns: Sampling, 
Data collection design, and Instrumentation. 

Goals of Study 
The follow-up design is based on the information needs 
of several audiences and focuses on a few broad system 
goals. Primarily, the impact evaluation of the CBVA 
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training is targeted at providing product developers with 
feedback as to how the field-test training impacted 
participants in terms of knowledge^ attitudes, and 
professional practices; and how that training impacted the 
total work environment* in addition, data gathered from 
the follow-up should provide insights to trainers in 
special education and vocational education personnel 
development programs who utilize the CBVA training 
materials developed through this project* Professionals 
directly involved in the use of curriculum-based assessment 
practices with special needs students in vocational 
programs, or administrative personnel responsible for the 
assessment policies and resource allocations at their 
schools, may also find the results of this study to be of 
assistance in local implementation efforts. 

Timelines for study 
Since the follow-up study had been clearly identified 
as a third-year project activity of the larger study in the 
original grant proposal, the related activities were 
structured within the larger project timeframe. Initial 
planning activities for the follow-up study commenced in 
June 1988, immediately following a May 1988 project staff 
laeeting in Washington, DC, during which a preliminary 
follow-up design outline was reviewed and suggestions were 
offered by the project director, site directors, and 
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trainers. Data collection^ which was to occur during three 
major on-site visits^ took place during October and 
November 1988* Data analysis occurred early in 1989/ and 
the final report/ including conclusions and recommenda- 
tions/ was completed in April 1989. A more detailed 
account of the study timelines is shown in Table 1. 

Major Operational Decisions 
Preliminary decisions about the follow-up design 
originated from four sources: the original project 
proposal/ a review of the related literature/ the May 1988 
project staff meeting in Washington DC, and input from the 
project director and other project personnel. A review of 
the related literature provided a context from which 
informed decisions about the design of the impact 
evaluation could be made. By examining key characteristics 
of the study (e.g./ audiences/ major system ^oalS/ 
resources/ evaluation settings/ and data needs)/ an 
appropriate methodological orientation was established. 
The study/ being exploratory in nature/ would yield the 
most useful data through a primarily qualitative approach. 
This conclusion was supported by research and verified by 
discussions with the Project Director/ Dr. Leonard 
Albright/ and Project Consultant/ Dr. Brian Cobb/ Associate 
Professor of Occupational Education/ Colorado State 
University. 
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Table 1. System Timelines 



Months 




Activity 


June 1 - Julv 31 


1 

X • 


xaenuiry Jropuxacl on 




2. 


Determine Data Needs 


August 1-31 


3. 


UKzWKzXKJjJ J.110 U L 1 'Iu6n 


September 1-10 


4. 


Field-test instrument 








September 10 - 30 


5. 


Orient the Respondents 






(Set up appointments) 


October 1 - November 30 


6. 


Collect Data 






(in-service visits. 






preservice phone 






interviews) 


December 1 - January 31 


7. 


Data Recording and 






Storing (word process- 






ing and coding) 


February 1 - February 31 


8. 


Data Analysis 


March 1 - April 30 


9. 


Use of Data 






(Report/Thesis) 



Note . Total timeframe for follow-up study (Project 
Year III): June 1988 - April 1989 
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The preliminary design of this study indicated that 
instrumentation would consist primarily of on-site 
observations, in-depth interviews, and telephone 
interviews. The follow-up population originally included 
all in-service trainees, and a purposive sampling of 
preservice trainees from all training sites. Data from the 
evaluation would be analyzed across sites and within sites. 

These basic features of the study were adapted as a 
result of continued analysis, input from project staff, and 
formative evaluation data collected through the training 
process. The resulting design includes a follow-up 
population of three in-service sites as planned, although 
the sites selected for on-site obvservation reflect an 
adjustment to the originel plan. A fourth in-service site, 
the Denver Career Education Center, became involved in the 
field-test as a result of a "spin off" from the preservice 
work being done at the University of Northern Colorado. 
The Denver site was included in the present study in lieu 
of another in-service site. North Orange County ROP in 
California. This decision was made by the project director 
after reviewing the in-service program at the California 
site, the formative field test evaluation data from this 
site, and a recommendation received from one external 
reviewer of the field test data. Also, the original plan 
has been adapted by limiting the preservice population to 
trainers only. Further rationale for the selection of 

ERJC 
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specific training sitei for the follow-up and to limit the 
preservice follow-up population is provided in a later 
section, Sampling . 

The follow-up study "start-up" activities included an 
initial contact with freld-test participants, thanking them 
for their participation in the project and informing them 
of their possible participation in the follow-up study. In 
addition, the May 1988 project staff meeting in Washington 
DC provided an opportunity for discussion among the 
project director, site directors, trainers, and the 
researcher about general follow-up activities, purposes, 
and project staff involvement in the follow-up. 

Follow-up Population 
The follow-up study population consisted of a sample 
of the personnel involved in the CBVA field-test training 
program. The total field-test training group included 72 
vocational and special services personnel at four in- 
service sites; 12 at Chisago Lakes School System (MN) , 12 
at Monroe Comprehensive High School (GA) , 13 at North 
Orange County Regional Occupational Program (CA) , ar. ii 35 
at the Denver Career Center, th« "spin-off" site. Another 
68 special education, vocational education, rehabilitation 
and guidance services professionals, who recieved pre- 
service training at six universities, were also included in 
the total training population. Those six universities were 
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Bemidji State University, the University of Georgia, 
California State University, Long Beach, George Washington 
University, University of Northern Colorado, and the 
University of Vermont. 

Nine prcject staff members, including site directors 
and trainers, the project director and najor project 
consultant were also included in the study population. A 
combined total of 149 field-test participants comprise the 
population from which the follow-up sample population was 
selected* Identifying information, such as names, 
addresses, phone numbers, professional titles, and place of 
work for each participant, was collected and entered into a 
database for use in follow-up activities. 

The sampling techniques used to select the follow-up 
population are included in Sampling ^ a subsection of 
Developing a Methodol oc;y> 

Data Needs 

The data needs of the impact evaluation were based on 
three primary study goals, as identified in the original 
project proposal (Albright and Cobb, 1985) : 

1. To determine the extent to which trainees have 
implemented curriculum-based assessment and planning 
procedures in their programs; 
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2* To determine the impact of these procedures in 
expanding services for special needs students in these 
vocational programs; and 

3. To determine trainee and trainer satisfaction with 
the training programs • 

These broad goals were broken down into more specific 
objectives Hased on the conceptual framework of the CBVA 
system* Objectives related to the first system goal, (use 
of CBVA procedures) were formulated from key system 
features, as identified by Albright and others 1988)* 

The purpose of the study was to determine impacts in 
four areas: ;i) implementation, (2) personnel roles, (3) 
system, and (4) training program. 

!• Implementation 

(a) the extent to which trainees implemented 
CBVA procedures at three critical phases: during selection 
and planning of a student's vocational program, during the 
student's enrollment in the program, and during the 
transition of the student from the program into productive 
employment or an alternative; 

(b) the extent to which trainees used assessment 
activities which were directly integrated into the 
curriculum of the student's vocational program; 

(c) the extent to which personnel were 
collaborating to conduct CBVA activities and utilizing the 
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results to ensure success for students in vocational 
programs; and 

(d) the extent to which monitoring techniques 
being used were direct/ criterion-referenced/ and 
performance-based . 

in addition, objectives focused on key personnel roles 
at each phase and in each assessment activity in an effort 
to gain insights as to how the training was implemented 
within the various personnel structures at each site. The 
study examined key system features from the perspectives of 
various personnel roles to determine: 

2. Personnel Roles 

(e) who was taking a lead role at each 
assessment phase / and for each assessment activity; 

(f) who was taking a collaborative role; 

(g) who was taking a minor role; 

(h) who seemed to be the overall key 
implementer ; and , 

(i) what was the rolij of administration in 
implementation. 

Objectives will also be directed at the system-wide impacts 
of training to address the second system goal and determine 
how CBVA procedures have expanded services for special 
needs students at these schools. The follow-up examined 
the total work environment to determine: 



ERLC 



52 

3. System 

(j) what changes had occurred as a result of 
CBVA training in terms of local policy; 

(k) what changes in organizational structure had 
occurred as a result of CBVA training; and, 

(1) what staff changes had been made as a 
result of CBVA training^ 

Finally, objectives will focus on participant feelings and 
attitudes about the training program. Three major questions 
about trainer and trainee satisfaction with the training 
process included: 

4m Training 

(m) Did the trainers and trainees have a 
generally positive or negative feeling about the CBVA 
training experience? 

(n) How did the training experience affect the 
professional development of the participants? and, 

(o) How did the CBVA training affect the 
participant's ideas about assessment or educating special 
needs students? 

Specific evaluation questions, based oa these 
objectives, were formulated and used in the design of 
interview schedules for instructors, administrators, and 
trainers. The questions were structured to allow 
respondents to describe current practices, relate back to 
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practices used prior to training, and to draw on their own 
perception whether or not changes were due to CBVA 
training* 

Methodology 

The follow-up methodology evolved out of an analysis 
of several interacting elements of the study; (a) system 
goalsr (b) time, personnel, and cost resources, (c) data 
needs; and (e) population size and geographic location 
were factors considered in the design of this impact 
evaluation. Consideration of these areas provided a basis 
upon which decisions were made about overall methodological 
orientation, study type, sampling techniques, 
instrumentation, and analysis. 

The study design was broadly based on one guiding 
premise: the overall intent of the study was to gain 
insight into the effects of training and the implementation 
process in order to improve CBVA training efforts in the 
future. The formulation of the data collection strategy, 
therefore, began with an examination of the features of the 
training process. 

The CBVA training was conducted at all sites as a 
field-test of system modules in draft form, Tho 
distinction between field-test training and actual training 
had important implications for the follow-up design. 
Field-test training means that while trainees received 
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instruction on CBVA concepts and practices, participated in 
group activities, and completed in-service and preservice 
assignments; they were simultaneously engaged in the 
evaluation of the field test version of the training 
system and providing feedback on the training experience. 

Because of this dual focus, in some cases training 
"took a back seat" to evaluation activities. In general, 
field-test training denotes that both training efforts and 
training content are in an experimental phase. 
Furthermore, participants and schools were only committed 
to involvement in the field-test and follow-up activities; 
future implementation was not a stipulated requirement 
of participation. Therefore, precise expectations about 
trainee implementation seemed to be unrealistic and rigid 
follow-up data collection procedures designed to measure 
levels of use against system objectives were considered 
inappropriate for the present study. 

Instead, the follow-up was aimed at uncovering inci- 
dental outcomes of training that contributed to the 
formative evaluation process. Participant feelings and 
attitudes, as well as knowledge and skills, were targeted 
in the impact evaluation; for information about changes in 
the thinking and practices of participants regarding CBVA 
concepts would be of prime importance to product developers 
and future trainers. 
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Another aspect of the training process which 
contributed to the follow-up design was the variety of 
training sites / backgrounds of participants , and structure 
of training involved in the field-test. Training sites 
ranged from a comprehensive high school to a regional 
occupational program for in-service training. Preservice 
participant groups ranged in professional backgrounds of 
rehabilitation services to special education. While in- 
service training involved all system modules, preservice 
training groups used only selected modules. Training 
strategies were experimental, with some trainers using 
structured, lecture and discussion formats and others using 
informal brainstorming techniques. For example, at one 
site, training occurred on Saturdays at the university, 
while at another site training was scheduled during the 
teacher free periods and lunch breaks. 

In essence, training was not a standardized process; 
the variety of sites and flexibility of training approaches 
were deliberately built into the field-test design. The 
logical conclusion was that standardized instrumentation 
such as surveys, checklists, or rating scales would not be 
useful in this study, in that they would not allow for the 
depth of data required to capture the unique conditions at 
each site. 

The follow-up study design was based on the assumption 
that a wide range of outcomes of training was possible and 
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data collection procedures needed to be flexible enough to 
accomodate that range. An in-depth look at participants in 
their work environments was necessary to provide meaningful 
impact evaluation data. An emergent strategy, in which 
patterns could develop instead of being preconceived, would 
allow for unanticipated outcomes to surface at each site. 

Sampling 

The follow-up study population sample was drawn from 
three sub-^populations including, 72 in-service trainees, 68 
preservice trainees, and 9 CBVA trainers. These groups 
represent four in-service training sites and six pre- 
service sites. While the original follow-up design 
included all in-service trainees at three sites (Monroe 
High School, Chisago Lakes High School, and Noith Orange 
County ROP), and a stratified sample of preservice 
trainees at each of the six universities involved in the 
field-test, this plan was altered in several ways. 

First, when field-test data indicated that the CBVA 
training system should be focused on use in in-service 
contexts, it was decided upon the recommendation of the 
Project Director and from the May, Washington DC 
meeting, that follow-up activities should concentrate 
mainly on the in-service population. Secondly, formative 
evaluation data collected during i:he field-test indicated 
that CBVA training at North Orange County ROP had been 
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problemmatic due to the logistics involved in an ROP 
setting, a change in key personnel at the site early in the 
training, and an unstructured, loosely scheduled training 
arrangement. Upon collaboration with the Project Director, 
it was decided to reduce follow-up at this site to aa 
interview with the Vocational Special Education Resource 
Teacher (VSERT) and the Administrator involved in the 
field-test training. 

As a result of these decisions, the sample of in- 
service sites studied in this follow-up included Monroe 
High School (GA) , Chisago Lakes High School (MN) , and 
Denver Career Education Center as a third follow-up site, 
with limited follow-up activities at North Orange County 
ROP. 

The particular circumstances surrounding the selection 
of Denver Career Center are significant in terms of the 
follow-up design, in-service training at Denver Career 
Center was an extension or "spin-off" of training 
activities at University of Northern Colorado (UNCO) • in 
fact, neither UNCO nor Denver Career Education Center was 
originally identified as one of the project's field sites. 

However, during the second project year, UNCO 
university faculty contacted the projec director 
expressing a great deal of interest in the project and 
requesting that UNCO be included in the field-test program. 
Formative data from the other field-test training sites 
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indicated that the CBVA process would also be particularly 
suited to student assessment needs at Denver Career Center 
and that project staff at this site were enthusiastic 
about field-testing the CBVA process at the Center* It 
was evident that impact evaluation data obtained at this 
site would be useful in providing information for future 
implementation efforts and so UNCO and the Denver Career 
Center were identified as part of the follow-up study 
sample. 

A second adaptation to the original plan involved the 
preservice follow-up sample. Instead of conducting 
interviews with a stratified sample of all preservice 
participants, the preservice follow-up sample was limited 
to site trainers and only at those preservice sites for 
which the locations would coincide with the in-service site 
visitation. Trainers at University of Georgia^ Bemidji 
State University, California State University, Long Beach, 
and University of Colorado would be interviewed during the 
in-service site visits. 

Within the in-service sites, purposive sampling was 
used to increase the efficiency of the data collection 
process during each site visit. Each in-service training 
program involved participants from five general personnel 
categories including, administrators, vocational/special 
education resource teachers, vocational instructors, 
special education teachers, and guidance counselors. The 
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CBVA training incorporated individual and group efforts on 
the part of these staff members and optimum use of the CBVA 
strategies depended to a great extent on the appropriate 
functions and collaborations of particular personnel at 
certain stages in the assessment process. For this reasonr 
the follow-up design focused on the perspectives of 
respondents by personnel categories, and a purposive 
sampling of representatives from each category was used. 
Collecting data from instructors, administrators, 
counselors, and the vocational special education resource 
teachers (VSERT) , at each site, would facilitate data 
analysis later, but would also serve as a cross-checking 
strategy for increasing validity of data received from the 
respondents. 

A second purposive sampling technique was used to 
maximize the quality of data at the in-service sites. A 
systematic approach to readjusting the sample as data were 
accumulated wes used to enhance the productivity of data 
collection. This strategy, known as "grapevine sampling" 
(Brinkerhoff and others 1983) involves a growing or 
changing sample. Interviewees supplied data that 
identified other participants as significant sources of 
follow-up data for future interviews. This sampling method 
was particularly useful in the present study since it was 
flexible enough for a wide range of training situations and 
efficient enough for a one-shot follow-up study. Since the 
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purpose of the study was to provide formative evaluation 
data to improve the CBVA training process in the future and 
not to make generalizations about the typical participant, 
this form of purposive sampling was preferable to random 
sampling techniques. 

In addition, another form of purposive sampling, known 
as key informant sampling (Brinkerhoff and others 1983), 
was used throughout the data collection process. By using 
this sampling technique, members of the population were 
selected who had the most information about certain 
conditions or situations. 

in summary, sampling techniques included various 
purposive methods in order to obtain rich data from a 
carefully selected population. By usinj the key informant 
sampling technique, training site coordinators were free to 
identify those who would participate in the follow-up based 
on their intiiriate knowledge of the work setting and 
training outcomes. The grapevine sampling technique was 
used on an on-going basis to continually adapt the follow- 
up sample as opportunities for quality data arose. 

Data Collection Design 

A narrative description of the general data collection 
design and rationale is i.ollowed by an example of a 
specific site evaluation plan in Table 2, Data Collection 
Design, Site 1. The table outlines a typical daily 
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Table 2. Data Collection Design — Site 1 
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Day Personnel 



Instrument 



1 Site Director 



Unstructured 
interview (use 
trainer schedule) 



Objectives 



Impact evaluation 
data: preservice 
and in-service 



Interview summary 
Key informant tally 



Overview of site^ 
directions re: 
key informants 



System 
Advocate 



Unstructured 
interview (use 
VSERT schedule) 



Impact evaluation 
data : in-service 



On-site observations 
tour of facilities 

Interview summary 
Key informant tally 
Observation summary 



Overview of site 
directions re: 
key informants, 
group interviews 



Selected Unstructured Impact evaluation 

groups as group interview data: in-service 

indicated use appropriate 

by previous schedules) 

data Interview summary 

Key informant tally 



System Advocate 



Key individu- 
als as indica- 
ted by previous 
deca 



Informal 
interview based 
on previous data 



Unstructured 
in-depth interview 
(use appropriate 
schedule) 



Review summaries 
for verification 

Target individu- 
als for interviews 

Specific case 
study evaluation 
data 



Note > Site: Chisago Lakes High School, Minnesota; 
Days on Site: 2. 
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itinerary at a follow-up site visitation and describes 
related instrumentation, personnel, and objectives. 

This qualitative approach to impact evaluation relied 
heavily on the purposive sampling techniques described in 
the preceding Sampling section, in addition to being used 
to select the follow-up sample population, key informant 
sampling techniques were used to focus and re-focus on key 
interview topics as part of an overall strategy designed to 
follow data patterns as they emerged. The flexibility of 
this approach allowed for unanticipated data sources as 
well as unanticipated training outcomes to surface. The 
emr^hasis on emergent methodology was in keeping with the 
exploratory nature of this study. 

Although the qualitative orientation of the design was 
intentionally flexible, it was based on a structured set of 
data needs and cross-checking procedures. For example, it 
was clear that the site directors (trainers) and system 
advocates (VSERTs) were two critical data sources. The 
trainers had been identified as the pre-service follow-up 
sample; they would provide data about pre-service training 
outcomes and assist in directing the in-service follow-up 
activities by acting as key informants. The system 
advocates were targeted as the coordinators within each 
school system during training; their role in system 
implementation was pivotal. The system advocates would 
further direct the follow-up interview schedule by 
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organizing appropriate group and individual interviews, 
interview settings, and time allocations based on their own 
expert understanding of the group relations and work 
setting* 

The primary data collection procedure used was the 
unstructured interview. Separate interview schedules, 
designed to collect data from each of the five personnel 
categories, were readily available to guide interviews by 
focusing on these various perspectives and on the defined 
system objectives. Data from these interviews were 
recorded and later summarized on Interview Summary Sheets . 
In addition, an instrument to record references to 
significant data sources, the Key Informant Tally , was used 
to substantiate the emergent theory and enhance analysis 
efforts later. Informal observations about the facilities, 
social interactions, and the general atmosphere during the 
visitation were recorded on an Observation Summary sheet , 
which was designed to collect contextual data. 

The design also included various check points to 
ensure the validity and reliability of the data as it was 
collected. First, the use of multiple sources within the 
primary data collection instrument, the interview, was an 
example of within-methods triangulation as described by 
Jick (1979). By using both interviews and on-site 
observation, across-method triangulate' on was used to verify 
data. Convergence of data from multiple sources about one 
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dimension of the impact evaluation increased confidence in 
the validity of the data as an indication cf system impact 
and not an artifact of the method used* 

Secondly/ by reviewing the interview sumip-'^ries with 
the interviewees/ misinterpretations and inaccuracies were 
discussed and corrected. Finally, a Site Analysis # 
occurred after ':he site visit with the Project Director, 
was used to review and evaluate the entire site visit. 
Data from this meeting was used to improve data collection 
during the upcoming site visit. 

Examples of the various intruments used in the data 
collection process are supplied in the following 
appendixes: Interview Summary Shr , Appendix A; Key 
Informant Tallv t Appendix B; Observation Sumiuarv Sheet s 
Appendix C; and Site Analysis # Appendix D. 

Instrumentation 

The study utilized four separate interview schedules 
for interviews with (a) trainers; (b) system advocates; 
(c) vocational and special education teachers and guidance 
counselors; and (d) administrators. Group interviews 
combined interview schedules as needed. Interview 
questions provided a general framework for the interviews, 
but the actual interviews were, in some cases, centered 
around selected topics depending on the respondent's 
experience with CBVA training or implementation. The 
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interviews were open-ended by design, to allow the 
respondent to freely supply what he or she felt was 
relevant information. The interview schedules by personnel 
type are shown in Appendixes E to H. 

Each set of site interviews was preceded by an 
Ov/erview of the CBVA process as it was presented during 
training, a bi.ief summary of changes made to the training 
products as a result of the field-test, and a statement 
about the purpose of the follow-up and use of follow-up 
data. A general description of the follow-up data 
collection procedures was included to invite participants 
to contribute to the process by suggesting additiona? data 
sources (e.g., programs* to observe, individuals to contact, 
example documents to review) throughout the visitation. 

The interviews of trainers were designed to collect 
data about the nature of the trainers' involvement in the 
field-test, and their perception of impacts of training on 
the in-service and pre-service participants. In addition, 
the interview focused on impacts of training on the 
proferssional activities of the trainers, i.e. continued use 
of CBVA in university courses, workshops, and research 
efforts. Finally, the trainers acted as key informants, 
supplying information about in-service trainees which 
served to direct data collection procedures at the in- 
service sites. These interviews were conducted in person 
and by telephone, when necessary. 
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The interviews of the system advocates at each site 
were de^Jigned to obtain an overall perspective on the 
field-test training experience and outcomes related to 
individuals and the total work environment. The system 
advocate, whose role at the schools enabled him or her to 
work with vocational instructors, special education 
teachers, guidance counselors, and administrators, was 
naturally in the position to observe changes in the working 
relationships, practices, and attitudes of the trainees. 
The system advocate, as coordinator of the CBVA program, 
was targeted for data collection about system-wide changes. 
Also, the system advocate was viewed as a prime source of 
key informant data. 

Interviews of administrators were designed to be brief 
and general in nature, as it was noted through formative 
evaluation data that administration at the three in-service 
sites was minimally involved in the field-test activities. 
They were interviewed, in some cases, to obtain key 
informant data, overall perceptions about impacts of 
training, and changes in policy, resource allocations, or 
staffing at the schools as a result of CBVA training. 

The interviews of vocational instructors, special 
education teachers, and guidance counselors were arranged 
according to categories that paralled the CBVA system as it 
was presented in training, focusing questions on either 
placement, environmental assessment, planning, monitoring. 
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or transitional services. An opening discussion was used 
to recall for the trainee the structure of the CBVA process 
in terms of these categOiTies. The interviews began with a 
broad question relating to the trainees overall feelings 
about the training program md continued with increasingly 
specific questions designed to encourage the respondent to 
describe training outcomes in detail. The patterning of 
the questioning was designed to first have the respondent 
describe current assessment practices, and then express 
their perception of the impact of training on these 
practices. Specific probes were used to assist trainees in 
their responses. 

Data Analysis 

In a qualitative study, data analysis is an on-going 
process. Analysis actually begins when the study questions 
are formulated since preliminary decisions about what kind 
of data will be collected are based on preconceptions about 
the outcomes of the study. Analysis continues through data 
collection; through the notetaking process some data are 
omitted, while other data are included and recorded in 
detail as the researcher decides what dimensions are more 
important and meaningful. Through the discretion of the 
researcher, this selective process shapes the data 
collection process and as analysis and data collection are 
interweaved, patterns emerge and theories are developed. 
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The qualitative data analysis is a creative, inductive 
endeavor guided by systematic checks for validity and 
repeated re-evaluation of theories (Miles and Hubermana 
1984) . 

Data analysis for this study began, then, with the 
data collection design* Sampling techniques, study 
questions, and instrumentation were carefully selected on 
the basis of a variety of training site characteristics. 
The data collection design was, therefore, flexible enough 
to allow for unanticipated outcomes. However, a 
preliminary analysis of field-test data, and input from the 
Project Director, Project Consultant, and site Trainers 
were utilized to focus data collection at each site on key 
questions, personnel, or instrumentation* In fact, prior 
to each visit the Project Director provided his perspective 
of the essential aspects of the particular training site 
and offered insights into interesting data sources or 
possibilities. Trainers were interviewed prior to the site 
visitation as well, not only for pre-service data 
collection but to aid in the pre-analysis process for the 
in-service data collection. Trainers gave their 
interpretation of the overall level of acceptance of the 
CBVA system, suggested probable productive data sources, 
and informed the researcher of what could reasonably be 
expected in terms of training outcomes. 

/ 7 
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The use of grapevine sampling was, in itself, an 
importc.nt part of the analysis process. The initial in- 
service site contact for the follow-up was with the site 
coordinator, who was advised to organize interview 
arrangements on the basis of his or her familiarity with 
the trainees, their work schedules and most importantly, 
their involvement with CBVA implementation since training. 
The actual interview schedule was adapted, in some cases, 
as data surfaced which indicated an adjustment would be 
mere productive. Interviews were shortened, lengthened, 
added, or omitted then, at the evaluator's discretion 
(although sometimes as a result of circumstances beyond the 
researcher's control). Also, based on information gained 
during an interview with a trainer or trainee, data 
gathered from another interview was weighted more heavily 
or lightly depending on the input. In this sense, data 
collection could hardly be separated from data analysis. 

The data analysis process could be summarized around 
three interactive ph-iises, as suggeshed by Miles and 
Hubermann (1984) : data reduction, data display, and data 
analysis. These phases are components of the data analysis 
which occurs after data collection. 

Data reduction for this study included several cycles. 
First, notes were reviewed and summarized with the 
interviewee for verification and completion. Secondly, the 
notes were more thoroughly reviewed by the researcher ard 
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details and ambiguous responses were clarified as 
recollected by the researcher. Next/ a crude coding system 
was developed and utilized to identify responses as either 
"general outcomes," "specific outcomes," "stray subjects," 
"anecdotal information," "observations," "descriptive 
information," and "key informant". These categories were 
used mainly as an organization tool at this stage since 
patterns as of yet were unknown. After all interviews at 
the site were completed, other coding categories, such as 
"emerging theory," and "methodology" were used as a total 
perspective was gained. 

Finally, a site analysis, a discussion with the 
Project Director served as yet another cycle of data 
reduction and review. As data were summariscc: and 
reported to the Project Director, he verified or questioned 
the emerging theories as they related to the data and/or 
offered alternative theories. Through this process, a 
framework for the following site visit was built, the 
coding system was refined, and data collection techniques 
were fine-tuned. 

Upon completion of the site visitations, the notes 
were transferred to instruments including the Interview 
Summary Sheet, the Observation Summary Sheet, the Document 
Summary Sheet, and the Key Informant. Tally Sheet. Samples 
of these instruments illustrate yet another cycle of data 
reduction. See Appendixes A-D. Here, the coding system 
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had been refined and is evidence of an advanced stage of 
analysis. Through the recycling of data from notes to data 
summary sheets, an overall sense of the data was acquired 
and it became easier to identify information "bins," a term 
used by Miles and Hubermann (1984) to describe the general 
constructs for discrete events, conditions, and behaviors 
in the data. 

The process of identifying bins could not easily be 
distinguished from another critical component of the data 
analysis process — the development conceptual framework. 
The development of a conceptual framework for the data 
analysis is essential even at the data reduction phase. 
The conceptual framework for thi-' study came out of two 
sources: the study objectives and the data collected. By 
comparing and contrasting the major facets of the study 
objectives with the general categories of data, a workable 
conceptual iramework was created. 

First, it became clear from reviewing data summaries 
of all sites that there seemed to have been various levels 
of implementation surfacing from training outcomes. On a 
heirarchical continuum of training outcomes, "improved 
communications among trainees" had emerged as an outcome on 
one end and "system-wide implementation" had emerged at the 
other end. During the pre-analysis phase, several emerging 
theorleL indicated that factors such as key advocate 
involvement, school climate, state level support, and 
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training strategies might be critical elements of the 
implementation process. By contrasting these factors with 
outcomes at the sites, a context for analysis of 
relationships was developed. 

The coding system, or bin labeling was derived from 
the study objectives as they are listed in Chapter 3, Data 
Needs f pages 7-10. The coding system, which is presented 
in Table 3, included labels for the data categories 
identified in the conceptual framework. After all data 
summary instruments had been coded, the data were word 
processed to a Data Summary Sheet, where data were 
organized by coded categories to facilitate analysis. An 
example of the Data Summary Sheet is shown in Appendix I. 
In summary, the conceptual framework was developed through 
the process of reducing data into bins, and refining the 
bin labeling through an analysis of study objectives and 
the emerging theories. The conceptual framework was built 
around a heirarchical continuum of outcomes and the 
identification of certain conditions at each site. 
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Table 3, Data Coding System 



Code 


Description 


COM 


Improved coitimunication among trainees 


AWARE 


Increased awarpnp*?*? a<?<?p<?<?Tnpnf" iqqmoc 


COLL 


Increased collaboration activities 


DA-PLACE 


Use of CBVA for placement 


DA-PLAN 


Use of CRVA fnr nlanm'nn 


DA-MON 


Use of CBVA for monitoring 


DA-TRANS 


Us*i of CBVA for transitional services 


TRAIN 


Satisfaction with training 


ADMIN 


Role of administration 


SYS 


Systemwide change 
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Chapter 4 
Results and Analysis of Data 

Introduction 
The analysis and sununary of data are closely 
integrated in a qualitative study* The selected method of 
organizing the data i:^ both a result of pre-analysis and a 
tool for facilitating continued data analysis. The data 
are summarized here according to categories of levels of 
implementation and implementation variables as they emerged 
through data reduction. Also, the data are summarized 
first within sites and secondly, across sites. Again, the 
summarization process mirrors the analysis process. 

The data summaries for each in-service site are 
organized according to the conceptual framework which was 
developed on the basis of data content areas and patterns 
as they emerged through a review of the data collected from 
all sources. The conceptual framework reduces data into 
two categories: implementation variables and levels of 
implementation. Implementation variables were identified 
as either (a) the role of the system advocate, (b) the work 

.ivironment, (c) external support, or (d) continued 
training activities. Levels of implementation wer« 
indentified as being related to either (a) increased 
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communication among trainees^ (b) increased awareness about 
assessment issues # (c) increased collaboration to conduct 
CBVA activities/ (d) use of CBVA practices within 
placement/ planning/ monitoring/ or transitional service 
functions/ (e) program modification and the use of support 
services as a result of CBVA and (f) system-wide 
implementation. Data relating to other areas/ such as role 
of the administration/ satisfaction with training/ and 
impact on professional development were included in the 
data summary as well. 

Data were coded/ regrouped/ tabulated/ and analyzed 
according to the methods described in Chapter 3/ Data 
Analysis . The data summaries are presented with as much 
consistency in format as possible; the conceptual frame- 
work/ as mentioned above/ provided a basic structure for 
the presentation of data. However/ the nature of a 
qualitative study/ relying heavily on open-unded interview 
formats/ grapevine sampling techniques/ and direct 
observations/ does not lend itself to neat, statistical 
data tabulation. AlGO/ the copious notes generated through 
multiple interviews and many days of observation cannot be 
concisely presented without losing much of their meaning. 

For this reason/ the within-site summaries are 
presented in varying degrees of detail. For example/ at 
Site 1/ Chisago Lakes High School/ the site visitation plan 
was structured in advance by the system advocate/ data 
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collection adhered closely to interview schedules and 
included all field-test participants^ Data from this site 
were easily reported according to source by personnel role^ 
frequency of response, and content areas as they were 
covered by the interview schedules, in contrast , follow-up 
activities at the Denver Career Education Center were 
planned to capture the impacts of training as they occurred 
at this site. This meant meeting with the "core group" of 
field-test participants instead of the entire group of 
trainees r and focusing on the unique set of events and 
conditions that led to system-~wide implemen'- ,tion at this 
site instead of the format imposed by the interview 
schedules. The data summary for this site, then, ic 
presented in a more narrative style with less attention to 
frequency of responses and date categories as they v;ere 
structured in the interview schedules. 

Separate preservice data surrimaries are also reported 
by site, highlighting significant outcomes and contextual 
data as described by the preservice sit.?, directors. Data 
gathered through preservice interviews were, in some cases, 
woven into the in-r.ervice data summaries, particularly when 
site directors provided key informant data about in-service 
training or were the source of triangulation data. 
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Presentation of Data—Within-site Analysis 
In-Service Site 1: Chisago Lakes High School 

Contextua.''. data for Site 1 centered around three major 
potential implementation factors: Work Environment, System 
Advocate, and External Support, in regard to work 
environment, the data gathered through interviews included 
repeated individual comments indicating that staff members 
felt pride in their staff and facilities. Of 14 
individuals who were interviewed, 5 made comments conveying 
a positive attitude towards their work environment, 
stating, for example, "We're pleased with the staff and 
facilities — everyone is very cooperative"; "This is an 
impressive group of vocational instructors — we're unique in 
this respect"; and "I feel that this school is progressive 
and despite other offers, I chose to stay here." These 
commeni^s were made by an academic instructor^ two 
vocational instructors, a special education teacher, and 
the vocational education resource teacher. These comments, 
incidently, were volunteered responses; there were no 
interview questions regarding the subject of the 
participant's feelings about the work environment • 

Several observations supported the finding that 
personnel at this site demonstrated a positive and 
cooperative attitude during the follow-up activities and 
that in general, school morale was high. All trainees 
participated in the follow-up activities and met first for 
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an early morning reception and orientation, readily 
volunteered information during an impromptu tour of the 
facilities, and each spent approximately one hour for 
individual interviews. Office personnel offered their 
assistance and support during the two-day follow-up visit 
and special arrangements had been made to schedule room 
space to be used during the follow-up activities. Also, 
the fact that involvement in the follow-up activities was 
optional and that all trainees agreed to participate at the 
VSERT's request, suggests a positive, cooperative attitude 
towards the follow-up activities. 

Several observations were made which indicated that 
the vocational education programs at the school were 
valued. The business program had received the highest 
local award — the "SPIRIT" award and the walls of the 
business progams office were covered with awards and 
letters of recognition. The VSERT had kept a scrapbook for 
the work <;xperience program which documented 15 years of 
community projects, newspaper clippings, award dinners, and 
student projects and accomplishments. Data collected on 
the Key Informant Tally Sheet, and through the interview 
process included many remarks about the Vocational 
Education Resource Teacher, who as coordinator for training 
activities at this site was identified as the "system 
advocate." she was repeatedly praised for her outstanding 
qualities: "she knows her job well"; "She's organized"; 
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"She's good about providing background on special needs 
students I have in my program"; "Her heart is there"; "She 
has the background and knowledge"; "She's a mover and 
shaker"; "She wears easily— she has a goori personality"; 
and "She can talk you into something nice?y — not 
aggressively" were comments made by vocational and special 
educators f counselors, and an academic instructor. These 
comments were not made in response to direct questions 
about the VSERT, but surfaced during the course of 
interviews* 

The third contextual area which emerged from the data 
was related to external support. During a site analysis 
meeting with the Project Director it was noted that in the 
state of Minnesota vocational education was a highly valued 
program, that this support was evident "from the capitol on 
down" and that vocational education was also a highly 
valued program at the University of Minnesota. In 
addition, data gathered through an interview with the Site 
Director, Dr. David Kingsbury, who load preservice training 
activities at Bemidji State University as well as in- 
service training at Chisago Lakes High School with the 
system advocate, indicated that the state had been active 
in developing the curriculum-based assessment concept for 
Minnesota Public Schools prior to the project field-test at 
Chisago Lakes. Likewise, the System Advocate mentioned 
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that support of vocational education at the state level had 
contributed significantly to uer career decisions* 

Data gathered through the interview process are 
sunmiarized according to levels of implementation as they 
emerged. These levels include: communication; awareness 
of assessment issues; collaboration; direci assessment 
activities; and program modification and support services* 
Other categorical areas of data include: satisfaction with 
training, role of administration, and professional 
development. 

During 14 individual interviews, 12 respondents, 
including 4 special education teachers, 4 vocational 
education teachers, 3 counselors, and the vocational 
education resource teacher indicated that an overall 
benefit of CBVA training was improved communication among 
staff members. These remarks were made in direct response 
to a question about the "most outstanding benefit of CBVA 
training." Comments described an overall improvement in 
communication: "It brought us together in one place — 
usually there is no time for this"; "It was critical that 
members got together and discussed — this (CBVA training) 
was a Vehicle for communication"; and "Now we have a more 
in-depth understanding of what is done in different 
vocational programs." 

Other comments indicated improved communication 
between specific professional roles: "There has always 
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been a general feeling among teachers that special 
education teachers have a lew student ratio — an easier job- 
-and there has been some apprehension about this • • . CBVA 
training has alleviated some of this"; and "Now they (VSERT 
and Vis) know what each other wants — they know what 
expectations are now." 

Frequent responses echoed the feeling among trainees 
that CBVA training had increased awareness of assessment 
issues and concepts as they relate to special needs 
students, in general, CBVA training was recognized as a 
"consciousness raising" activity and more specifically, 17 
respondents made comments related to awareness of 
assessment issues. These occurred in three areas: 

1. Awareness of current (pre-CBVA) assessment 
practices and how to improve upon them, 

2. Awareness of deficiencies in present assessment 
practices (either during placement, planning, monitoring, 
or transitional services), and 

3* Changed perceptions about the purpose or nature of 
assessment. 

Couiments made by special education teachers, 
vocational instructors, the vocational education resource 
teacher/ and counselors were not in response to any 
specific interview question. Comments about improved 
awareness of current assessment practices included: "I'm 
more aware of the assessment procedures being used by 
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counselors and am better informed on how to use them with 
my (special education) students*" Increasesd awareness 
about deficiencies in present assessment practices was 
expressed in such comments as: "We would like to see more 
help in the area of screening — we need help with looking at 
curriculum and making adjustments for special needs 
students — with the ecological assessments." 

Comments related to a changed perception about the 
purpose of assessment included: "I now use assessment, not 
as a weeding toolr but for identifying weaknesses." 

Interview responses indicated that trainees were 
collaborating to provide assessment and support services 
for special needs students. Specifically, collaboration 
efforts were noted in six practices: 

1. Vocational instructors are providing special 
education teachers with information about their programs to 
be used for placement and planning assessment activities; 

2. VocatioHcil counselor is visiting classrooms to 
become more familiar with vocational programs to aid in 
assessment activities for plr.cement; 

3. Vocational counselors are collaborating with VSERT 
for registration of speci'il needs students; 

4. Vocational instructors are collaborating with 
VSERT to provide support services for special needs 
students; 
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5. Vocational instructors are taking part in the lEP 
process; and 

6. Vocational instructors are collaborating with 
VSE^*^ to create assessment tools ^nd make program 
modifications • 

Nine responses indicated that trainees were using 
assessment activities during placement/ planning/ and 
transitional services and that these activities were 
directly integrated into the curriculum of the student's 
vocational program. Responses about direct assessment 
practices included those from four vocational instructors/ 
four special education teachers/ end the vocational 
education resource ieacher and were in direct response to 
questions about the use of direct assessment practices 
during the three phases of assessment. 

Comments regarding direct assessmenl activities used 
during placement and planning included/ "I developed a form 
for communication with parents which I use at lEP meetings. 
I^ve gotten some very interesting information from parents 
which- helps me with planning/" and examples of direct 
assessment activities described during interviews included 
a general automotive background assessment tool for entry- 
level skill assessment and program planning developed by a 
vocational instructor/ a slide interest battery developed 
by the VSERT to assess student interest for placement in 
vocational programs/ an alternative math program for 



special needs students developed as a result of thv^ 
assessment of all vocational programs for required math 
skills for entry, and an equipment identification 
assessment instrument developed by a vocational instructor 
to assess entry-level skills • 

One comment related to the use of direct assessment 
practices for transitional services assessment: "Training 
helped me formulate better questions for social service 
agencies, like the Department of Vocational Services, and I 
found out more about the Day Activity Center (sheltered 
workshop) 

Interview data indicated that trainees are using CBVA 
data to irake program modifications and to provide support 
services to ensure the success of special needs students in 
their programs* Vocational instructors reported having 
developed a number of strategies for adapting curriculum 
and instruction such as the use of teacher aides, peer 
tutoring, and special education staff to provide support 
services for special needs ::jtudents in their programs* 
Eleven examples of such modifications and support services 
were mentioned during interviews with ten vocational 
instructors and a special education teacher. 

In response to a direct question about satisfaction 
with CBVA training, comments were generally made in terms 
of specific benefits fo training and so were categorized as 
they related to previously noted outcomes* Respondents 
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generally described satisfaction with training in terms of 
improved communication among participants of the training 
program and increased awareness about the assessment of 
special needs students. However^ two trainees (an 
academic/vocational instructor and the VSERT) expressed 
their positive reactions to the training process^ 
perceiving it as a viable system of assessment and noted 
several benefits of the training. The academic instructor 
(fomerly a vocational instructor) felt the system had great 
possibilities, in that it avoided duplication of efforts, 
increased efficiency after start-up time, allowed teachers 
to keep closer tabs oii students, allowed for more 
interventions, quicker and more vocational involvement in 
the lEP process, and more involvement of parents. The 
VSERT felt that CBVA is a "common ground — a much needed 
tool — and a great start for changes." After 15 years in 
the field she felt that she had seen the need for CBVA a 
long time ago. 

Two specific comments were made indicating a skeptical 
attitude or ambivalent feeling about CBVA training. Two 
vocational instructors commented that they had been 
skeptical during training and felt that some of the CBVA 
practices v/ould not be feasible. Both stated, however, 
during follow-ap interviews that, overall, training had 
been a positive experience for the staff. 
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A few comments were made in direct response to 
interview questions about administrative involvement in 
training activities and they consistently indicated that 
there was little or no adrainistrative involvement in CBVA 
training since the school was "in between superintendents 
during the field-test." However, it was also mentioned by 
a special educator and a vocational instructor that 
administration was generally "supportive of innovative 
projects'" and were "willing to make changes if they saw a 
need. 

Two responses indicated that participants felt that 
training had impacted their professional development by 
either expanding their role or stimulating them to pursue 
further professional development in special education or 
assessment or special needs students. Namely, the VSERT 
saw her role as expanding since CBVA training, specifically 
in the area of transitional services, although she could 
not say for sure that this was a direct result of training. 
A vocational instructor did feel that her interest in 
special needs assessment was stimulated by CBVA training 
and that she was seriously considering graduate work in 
this area. 

A secondary outcome of training, the impact of the 
follow-up activities, was noted during the exit interview 
with the system advocate. When asked by her principal, 
"How did it go?" the system advocate was enthusiastic 



about the effect of the follow-up visitation* she 
described the follow-up as a "shot in the arm" and said, 
"It got things humming again*" She went on to explain that 
it had much the same effect on the staff as the 
accreditation proces/ boosting morale by giving the staff 
some well-deserved recognition and by "giving the teachers 
the opportunity to talk about their programs*" Two other 
trainees r one vocational instructor and one special 
education teacher , remarked that the follow-up visitation 
had stimulated them to follow through on some major 
assessment activities that were in the development ou^v,-!?. 

In summary, the d ca collected at Chisago Lakes High 
School through interviews, document review, and 
observations indicated that outcomes of CBVA training were 
primarily in the areas of improved communication; increased 
awareness of assessment issues as they relate to special 
needs students; collaboration among vocational, special 
education, and the vocational resource education teacher to 
use CBVAo The use of direct assessment practices during 
placement, planning, anu transitional services; 
modifications to vocational programs; and support services 
to ensure the success of special needs students in 
vocational programs were also outcomes that occured as a 
result of training. In addition, the data included 
contextual aspects of the site which indicated that the 
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work environment was positive and progressive, that the 
system advocate {the vocational education resource teacher) 
was a highly regarded, influential individual, and that 
external support from the state of Minnesota was present. 
The effect of the follow-up as an important intervention 
was noted by the system advocate and two trainees as well. 

Preservice Site 1: Bemidii State University 
A telephone interview with the site director at 
Bemidji State University covered the events leading up to 
his involvement in the field-test program, as well as his 
current involvement with CBVA and future plans for CBVA 
related work. The site director. Dr. David Kingsbury, 
acted as both preservice trainer at the university and as 
the director for training activities at the in-service 
site, Chisago Lakes High School. Training activities 
strictly related to the field-test program included one 
course for in-service trainees held at the high school 
campus, and one informal attempt at a pre-service course 
and two "official" preservice couses at Bemidji State 
University. 

The site director, however, described his involvement 
with the CBVA concept prior to field-test training as a 
long-standing interest in the work done in this area by 
Albright, Sarkees, Cobb, and Larkin. Describing his 
familiarity with CBVA as "80% conceptualized, he felt that 



project related papers prepared by Albright and Cobb 
crystalized his understanding of how the system could be 
used in school s« 

Others examples of involvement in CBVA activities 
prior to the Chisago Lakes field-test program included 
participation in state workshop which reflected attempts by 
the state of Minnesota to implement general CBVA concepts 
in Minnesota Public Schools • In addition, he had been 
approached by the State Department to prepare a vocational 
assessment manual for Minnesota, which he developed on the 
basis of his advocacy of the CBVA system. The site 
director explained that his involvement with the state 
workshops and policy development could be considered a 
"quasi -direct" outcome of CBVA training, since these 
activities began prior to official field-test involvement 
but were continued with the field-test activities serving 
as a catalyst to further involvement at the state level* 

The Chisago Lakes High School in-service training 
program, was delivered through the use of the CBVA project 
training modules and was described by the trainer as a 
positive, effective experience. He mentioned the superior 
leadership ablilities of the system advocate at the site 
and credited her for the successful reception of the field- 
test program and continued use of CBVA at the site* He 
felt, overall, that trainess had benefited from the field- 
test programs, but that although the interaction had 
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successfully introduced the content of the CBVA systemr 
organizational change would require at least one to three 
more years of extensive interventions. He commented on the 
meaning of "school-wide change," logistical change," and 
"functional change" and described the "push/pull effect" as 
a process through which implementation occurs in an 
organization. He explained that in order for the 
"technological transfer" to occur in schools, the change 
would have to meet ihe local organizational demands, in 
short, "the school would have to want it." 

The site director discussed another aspect of his 
pcirsonal involvement with CBVA concepts vhich has developed 
in a parallel fashion to his academic application. His 
work in the private sector in human resource development 
has included consultant services to three companies using 
CBVA principles for recruiting, hiring, and training people 
for jobs in manufacturing and production. He described 
this application of CBVA as a "systematic way of getting 
the right people to apply for jobs." He explained that the 
use of CBVA concepts for private industry had been 
developed independently by the owner of a paper 
manufacturing company, who had stimulated his interest in 
this kind of application. 

The site director expressed his intention to continue 
work in organizational development with thirteen school 
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districts in Minnesota and to start new programs using CBVA 
concepts, in summary , the site director's involvement with 
CBVA began before his participation as site director for 
the field-test program and will continue both at the 
university, in Minnesota public schools, and through 
consultant work with private industry. The trainer 
expressed his satisfaction with the field-test program and 
strong interebc in the further imp.lep<intation of CBVA 
concepts. 

In-Service Site 2; Monroe High School 
Contextual data for Site 2 related to the areas of 
System Advocate, External Support, and Work Enivornment. 
The system advocate, the vocatioanl education resource 
teacher (called Resource Vocational Instructor at this 
site) was repeatedly described during interviews as a 
highly valued colleague. The three vocational instructors 
interviewed and a counselor remarked that he was "a warm 
and caring teacher", "really good at his job", and that he 
was a great improvement on the individual who previously 
had the RVI position. It was apparent that the system 
advocate was quite familiar with the vocational programs 
visited for interviews, since he took over the classess 
while the vocational instructors were being interviewed. 
Also, during observation of the system advocate at work 
with his special needs students for approximately one hour. 
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he helped students with resumes and job counseling^ 
assisted a student taking a cosmetology test, and discussed 
plans for an extra-curricular activity with in-coming 
students; he handled a variety of student needs in a short 
time (in an extremely small office space) with ease* His 
relationship with the students appeared to be extremely 
good; it was obvious that they respected and liked him a 
great deal. Several awards hung on the walls of his 
officer among them three separate "Teahcer of the Year" 
awards. During the two day visit at the sice, the system 
advocate exhibited a high energy level / a positive attitude 
and a strong commitment to his work. 

During an interview, the system advocate described his 
influence over the participants in the fie]d-test and 
stated that the trainees were generally cooperative about 
training activities but considered it "just a test" and 
seemed to "go back to their old ways" soon after training. 
He added that if he had "pushed it", he could have 
influenced them to continue efforts to implement CBVA more 
formally, but that he had lost his enthusiasm for the 
project when shortly after the field-test program, the site 
director from the university resigned and moved out-of- 
state to assume another university post. 

This information seems to be significant since it 
verifies the critical role of system advocate and also 
since it aludes to the importance of external, in this case 
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university personnel, support for implementation. The 
system advocate summarized this notion by stating, "^en 
(Site Director left, i felt the project was unimportant . - 
He went on to say that since there was no immediate follow- 
up training, efforts at implementation were mostly 
discontinued. The idea of the significance of the loss of 
external support at this site, was verified through a prior 
interview with the site Trainer. Also, two interviewees 
had mentioned they appreciated the external support they 
received from university personnel during training. 

There were no specific data that indicated that this 
was a particularly positive or negative work environments- 
no comments were made during the visit to indicate that the 
staff was either enthusiastic or disillusioned with the 
follow-up visit, it was noted that preparations for the 
visit appeared to be minimal and only 5 of the 12 field- 
test participants were scheduled by the vocational 
education resource teacher for follow-up interviews. 

Decriptive data about the work environment, however, 
did included one vocational instructor's comments about a 
certain degree of tension that existed at the site due to a 
lawsuit pending in regard to a special needs student who 
had died as a result of an injury which occurred during 
incident and other similar problems al school in the area, 
he described, had instilled a fearful attitude among 
teachers about getting involved with special needs 
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students. He went so far as to admit that "a lot of 
teachers didn't want special needs kids in their classes." 

The s^me vocational instructor also described what he 
perceived as a negative attitude from parents and the 
community about special needs students. He decribed a 
tendancy towards overprotectiveness on the part of parents 
and sometimes a reluctance to encourage employment for 
their children because it would mean they'd loose monetary 
benefits. He also described high racial tension at the 
school, which had about a 50:50 black to white ratio, and 
that the community tended to "blow up" over school 
problems. These data could be significant in terms of 
viewing training outcomes. 

In summary, contextual data describing the role of the 
system advocate, the role of external support, and the work 
environment can be used to interpret interview data 
regarding levels of implementation at this site. Interview 
data indicated that trainees at Monroe High School had 
experienced improved awareness about assessment issues as 
they relate to special needs students; were collaborating 
with the VSERT for assessment; were using some direct 
assessment activities during placement, planning and 
transitional services; and were modifying instruction and 
providing support services for special needs students in 
their programs. 
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There were 23 responses made during interviews that 
indicated that there was an increased awareness about 
assessment of special needs students* Eight responses were 
from one counselor, including: "Before training my concept 
of assessment was Iowa tests — now I see it more as matching 
talents to special needs"; "I still see needs not being 
metr but awareness about special needs kids has increased"; 
Responses from three vocational instructors included: 
"I've always been aware of special needs students' 
different levels r but I've made some improvements since 
CBVA"; and "You can throw standardized tests out the 
window — students are competing against things they don't 
know about." Comments made by the vocational education 
resource teacher regarding awareness of assessment issues 
included: "The greatest benefit has been more awareness 
about options for special needs students"; and "I see the 
need for CBVA, after training, I took APTICOM and threw it 
in the trash." Three responses during interviews indicated 
that trainees v;ere collaborating to assess special needs 
students. Two vocational instructors and one counselor 
mentioned that they worked with the VSERT to adapt tests 
for special needs students and for transitional services 
planning. The counselor indicated that she relied on the 
VSERT for placement of special needs students in vocational 
programs. As mentioned in the observations about the 
system advocate (VSERT) , it was obvious that he was quite 
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familiar with many of the vocational programs and he 
asserted that he spent a great deal of time actually in the 
classrooms with each of his 25 special needs students. 

Interviewees gave seven examples of how they were 
using direct assessment activities for placement and 
planning of special needs students' programs as well as for 
transitional services for their special needs students. A 
vocational instructor described how he intervened in the 
case of a student who had been employed by a local lumber 
company after completion of his program: "He called me up 
(the student) and told me he was fixin' to quit because he 
was afraid to drive a truck — so I called the employer^ 
talked to them about it . . . and they said, 'no problem'. 
Ee still has that job." Another instructor uses a right- 
and left-handed threading nut and bolt test to determine 
mechanical ability and problem solving skills for making 
instructional decisions. He added, "I don't call it a 
test, I just say, 'do this', and I observe." Another 
instructor developed an interview and questionnaire for 
entry level skill assessment during training and now uses 
an adapted version. A vocational instructor described how 
she helped get a student a job in a veterinary clinic by 
preparing the employer that the student had special needs 
and made several follow-up visits to the job site. She 
added that the student was still employed after 2-1/2 
years. 
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Three examples of vocational instructors making 
program modifications and providing support services 
included a vocational instructor who now modifies his 
program for special needs students by assigning lower level 
projects/ a vocational instructor who now uses a flannel 
board identification test for fruit tree leaves instead of 
the 200 question agriculture and science test she 
previously used, and a vocational instructor who provided 
additional early morning instruction to demonstrate the use 
of appropriate drill bits for a special needs student who 
had reading difficulties. 

The data also included remarks about satisfaction with 
training from two trainees who described hov^ they felt 
training was lacking and suggestions on how they would 
improve future training efforts. A special education 
teacher felt that training was "borderline"/ that it 
"touched upon things, but didn't get into what you actually 
do with kids," and that it was "tough even fo^ someone with 
special education and resource vocational teacher 
background — for the typical vocational instructor it was 
too much — you need five times more training to really 
understand it." He compared training to introduction to 
Special Education and said that it didn't prepare teachers 
for the real situation. He made several other negative 
comments about the involvement of the counselors, "they 
just give lip service, they don't really get involved," and 
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the administration^ "they're not really involved — they just 
viewed it as only a test and a way to look good in the 
newspapers. . .the actual paper--work is another story." 
His suggestions for improvement on training and/or 
implementation included using a daily monitor or support 
person and at least a month for training. The VSERT had 
very specific ideas about how he envisioned improved CBVA 
training. He advised not to use modules for training but 
to only give an overview of concepts to trainees. He felt 
that instead of one--shot training, an on-going approach to 
training, with frequent follow-up, would be more 
effective. Also, he saw the benefit of using the field- 
test people as trainers because, "teachers relate to 
teachers — not bureaucratic projects." 

The system advocate also expressed his feeling that 
the follow-up visitation had renewed his interest in the 
project and reaffirmed his belief that the CBVA system 
could successfully be implemented at Monroe High School. 
He indicated that he would pursue the opportunity to 
participate in the up-coming training workshop to be held 
at California State University, Long Beach, utilizing local 
funds that he had access to by including several other 
field-test participants in continued training as well. He 
mentioned that he felt the trainees were encouraged by the 
validating effect of the follow-up visit. 
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Closing comments by the VSERT included his uncertainty 
about whether or not vocational instructors "really got the 
concept of CBVA," his feeling that "the vocational 
instructors still don't realize the importance of 
assessment," but that prior to CBVA they had no knowledge 
of the Carl Perkins Act and that "CBVA, overall is a 
fantastic program — we need full-fledged implementation." 

In summary, follow-up data indicated that outcomes of 
training at Site 2 were primarily in the areas of increased 
awareness of assessment issues; the use of direct assess- 
ment activities for planning, placement and transitional 
services; program modification and collaboration between 
the vocational instructors and the vocational education 
resource teacher. However, the limited number of field- 
test trainees who participated in the follow-up activities 
(five) suggested that training had not impacted the non- 
participating trainees (7) to the same extent. Contextual 
data indicated that the system advocate was an effective 
motivator and highly regarded at the site, and the external 
support from the university had been high but was withdrawn 
shortly after field-testing. The work environment was 
characterized as racially tense, and the general attitude 
about special needs students had been described as negative 
as a result of a pending lawsuit involving a special needs 
student, and the difficulties of parental and con.munity 
resistance towards improving services to special needs 
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students. Despite these conditions^ evidence also was 
obtained which suggests that the trainees had become more 
open to accepting and v/orking v^ith special needs students 
in their programs. Finally/ the follow-up visit was 
perceived as an important intervention for renewing 
interest in the CBVA system* 

Pxes.ervice_S.ite 2; Un iversity of Georgia 
Data collected during a telephone interview prior to 
and after the in-service site visitation with the trainer, 
focused mainly on background information about the in- 
service training program. Only brief mention was made in 
regard to the preservice courses taught at the University 
of Georgia by the site director and trainer, where the CB\ A 
system modules had been infused as the assessment component 
of a special needs course. The in-service training was 
delivered initially by the site director and the trainer at 
the University of Georgia campus where they jointly "warmed 
up" the trainees through two all-day sessions. Training 
continued at the high school during afternoon sessions, 
with the trainer providing technical assistance to the 
trainees on an individual consultant basis throughout the 
spring and fall 1987 training periods. 

The trainer discussed several important issues related 
to follow-up data collection which entailed key informant 
information, his perspective on the breakdown of support 
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that occurred during the field-test, and the role of 
administration in the field-test program. Interestingly, 
without having discussed plans for the follow-up activities 
with the system advocate at the site, the trainer 
identified four participants, in addition to the system 
advocate, whose "minds were ripe" during the field-test 
training and who would probably be the most productive 
sources of follow-up data. These four individuals, all 
vocational instructors, were in fact the participants 
recommended by the system advocate for interviews as well. 
The trainer felt that of the 15 vocational programs at the 
comprehensive high school, only the instructors from these 
four programs had "really gotten involved" with CBVA 
training. He also described a fifth member of the training 
group v/hoHi he recommended for an interview with 
reservations, commenting that this special education 
teacher would probably provide negative feedback. 

Secondlyr the trainer expressed doubt that the system 
advocate had continued efforts at implementing CBVA, 
perceiving the abrupt departure of the site director from 
the field-test training effort as a significant cause. The 
trainer perceived the influence of the "outside system 
advocate," the site director, as having been crucial to 
creating a system change. 
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Finally/ the trainer commented on the lack of 
administrative support at this site, but qualified this 
observation by noting that in a sense, "by giving full 
support to [the system advocate], administration was 
supportive." He also pointed out that the administration 
at the site did support the program by sending the system 
advocate to a Washington, DC project staff meeting. 

In-Service Site 3; Denver Career 
jBduc ation Center 

Contextual data collected at the Denver Career 
Education Center (DCEC) related to four major areas: 
System Advocate, VJork Environment, External Support, and 
Training Strategy. Data gathered through interviews, 
observations, and document review indicated that the most 
significant training outcome was system-wide implementation 
of CBVA at the Career Center and continuing implementation 
efforts throughout the Denver public school system. 
Impleipentation also occurred at the levels of improved 
communication, increased awareness of assessment issues, 
and collaboration. Data collection efforts concentrated 
mainly on the conditions, strategies, and elements which 
contributed to the successful implementation at the site. 

The role of the system aJIvOcate was repeatedly 
acknowledged in the data collection process as being of 
prime importance to the successful implementation at the 
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site* A pre-site interview with the Site Director 
(university personnel) indicated that the system advocate 
had njade "all the difference in the world for implementa- 
tion • • • she cleared the path • . . presenting CBVA as a 
viable form of assessment." She went on to describe the 
system advocate as being "the best thing that ever happened 
to our system." 

The system advocate was repeatedly praised during the 
group interview for being "highly respected in the field" 
and it was noted by the resource specialists that she 
"paved the way" for CBVA implementation by continually 
encouraging and motivating the trainees. Her effectiveness 
at briefing the principal and assistant principal on the 
federal mandates regarding assessment of special needs 
students and clearly relating these to the goals of the 
CBVA system was noted as a key factor in winning 
administrative support for implementation. The Site 
Director reiterated the unique talents of the system 
advocate in public relations, effective communication 
techniques, and people management skills. As the 
Supplemental Services Coordinator for Denver Public 
Schools, this individual's main function, to appropriate 
funds for training, supplies, and equipment to address the 
Carl Perkins Act, was perfectly suited to the role of 
system advocate for CBVA training and implementation. 
Likewise, by supplying funds for on-going training. 
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including summer development, the system advocate was able 
to maintain continuity from field-test training through 
implementat: on and as noted by the Site Director/ "will 
keep things alive." 

The role of the system advocate at this site was 
clearly critical; however/ it was the combination of 
several key factors that resulted in full-scale 
implemenation of CBVA training. A supportive work 
environment .'eemed to have fostered implemenation efforts 
as well. The assistant principal remarked that "CBVA goes 
perfectly with the Denver Career Education Center 
philosophy because it is a system in which all students 
find success." Other data gathered through the group 
interview included comments depicting the DCEC 
administration as highly innovative: "In the past three 
years/ we've had 21 new programs . . . anything neW/ 
exciting in schools/ we like to try." Another comment 
described the center prior to the hiring of the system 
advocate and current assistant principal as "the castle on 
the hill . . . you couldn't get special needs people in 
here." A resource specialist compared DCEC to a school she 
previously worked at saying that/ "It (former school) had 
no cohesiveness/ it was departmentalized/ territorial . . . 
here we all work together for the outcome of the school." 

Other comments relevant to the work environment at the 
site indicated that key DCEC personnel/ namely the system 
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advocate, was familiar with the concept of curriculum-based 
assessment prior to CBVA training: "Prior to training, I 
(system advocate) had looked at CBVA as one of the 
assessment techniques out there. The site director at 
University of Northern Colorado didn't have the training 
materials at that point, but when she did bring those 
materials in, we had already been moving on CBVA." The 
site director verified this prior activity in curriculum- 
based assessment remarking several times that DCEC was "so 
far ahead of the other schools (in the Denver Public School 
system) that training was used to refine and build on what 
they already had." 

Certain ^ther data related both to the work 
environment and the approach to field-test training and 
continued training suggested that the cooperative 
interaction of core personnel was critical for successful 
implementation. The core group, or training team, 
consisted of the site director from the university with 
another university faculty member lending full support, the 
supplemental services coordinator, the director of special 
services at the site, the assistant vice principal, three 
resource specialists, and one vocational instructor. 
Through this interagency involvement, a "pyramid" approach 
to training occurred that was described as a way of 
"sharing the load", and "getting others to buy into the 
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system." The site director stated that "it took all of us 
working together to pull it (CBVA training) off." Factors 
contributing to successful, full-scale implementation at 
DCEC included the role of the system advocate; a supportive 
work enviroment which involved external support from the 
university and the cooperative efforts of separately funded 
staff members; and a unique approach to field-test training 
and continued training that relied heavily on the first two 
contributing factors. Given a site already effectively 
working on curriculum-based assessment, a cooperative and 
motivated staff, a highly qualified system advocate with 
sufficient resources and influence over administration to 
initiate the system change, the site director found the 
DCEC to be uniquely suited to CBVA implementation. The 
training process was collectively described by the core 
group members as the following sequence of events: 

1. Prior to CBVA training, the system advocate had 
hired resource specialists to run the supplemental services 
programs at each of the Denver Public Schools. She 
carefully enlisted the support of the principal and 
assistant principal at the on-set to reinforce the 
relationship between the resource specialists and the 
administration, thus encouraging the acclimatization of the 
resource specialists to the school environments. 



2. The site director met with system advocate and 
director of special services at DCEC and informally 
reviewed the training materials. 

3. The three looked at assessment priorities at DCEC 
and identified the areas to be targeted in training as 
placement and planning, focusing on pieces of the relevant 
CBVA modules such as job try-outs and work samples to 
develop in the training. 

4. The system advocate recognized that the resource 
specialists were fully aware of the wide array of 
assessment practices currently available and was sensitive 
to their suspicion of change when she introduced the CBVA 
concept. 

5. The site director provided technical assistance 
on an individual basis to the three core group resource 
specialists, seizing an opportunity to work with one 
resource specialist who was having difficulty with a 
student inadequately placed in a vocational program 
according to an APTICOM assessment. 

6. The site director did not use actual training 
modules, but instead presented the process to the resource 
specialist by having her identify 10 critical entry level 
skills for predicting the student's success in the 
vocational program. After narrowing them down to five, the 
resource specialist was asked to create assessment tools 
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using those skills. This process became known as 
developing "job try-outs" or "modules" and was adopted as 
the on-going method of training of resource specialists in 
curriculum-based assessment. 

7. The initial "job try-out" or "module" was 
developed as a result of a joint effort on the part of the 
site director/ the director of special services? and the 
resource specialist who together attended the vocational 
class several times to observe first-hand the critical 
skill areas needed for entry into the coimnerical design 
class to be used for assessment. The vocational instructor 
worked closely with the group as well, in a process of 
breaking down tasks into specific skills and creating 
appropriate placement work samples. 

8. Once the first resource specialist had 
successfully developed and used the module, she joined the 
core group and proceeded to train two other resource 
specialists at the site. 

9. The system advocate continued to have monthly 
meetings with the entire group of resource specialists 
(approximately 30), alluding to the CBVA process but giving 
no formal training at this point except to assign the skill 
analysis task to the trainees as preparation for an up- 
coming training activity. (Training was to take place in 
Vail, Colorado and was presented as a "perk," an "exciting 



117 



109 

activity" and a working retreat." The system advocate had 
sufficient funds to provide accomodations for the weekend 
training activity including expenses plus fringe benefits 
for the core training group.) 

10. The site director and support university faculty 
member led the training seminar with examples of informal 
math and reading tools (not CBVA modules) to show the 
"process" of CBVA. 

11. The other core group members, which included the 
director of special services, three resource specialists 
and the vocational instructor, presented the concepts as 
they learned them through their actual use of CBVA at DCEC. 

12. The remainder of the two-day session consisted of 
a problem-solving, one-on-one workshop activity in which 
participants created work samples to be used as assessment 
tools for various vocational programs. 

13. Continued training occurred at the regular 
monthly meetings of resource specialists during which 
participants shared their experiences with using the CBVA 
assessment modules at their schools and new members 
developed try-out modules using a iitodel format. 

In addition to the group meeting that included the 
core training group members, the follow-up visitation 
included an observation of a meeting of Denver Public 
School resource specialists. This meeting had been 
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deliberately scheduled by the system advocate to contribute 
to the follow-up effort and allowed for an opportunity to 
collect data from the resource specialists about their 
satisfaction with training and their experiences with using 
CBVA* 

In general r the resource specialists felt "more 
positive than negative" about CBVA training, remarking that 
they felt "a bit overwhelmed at first • • • found it 
difficult to understand" but that later "after talking with 
instructors • • . saw a common thread, saw patterns, and it 
made sense." In a later interview, the site director 
stated that she felt the resource specialists appreciated 
being involved with the field-test, as it brought the 
recognition that comes from being associated with a model 
site and "being connected with something innovative." A 
few core group members had even thanked her for "letting 
them do this" and said that they were "honored" to have 
been a part of the project. 

Several comments were made suggesting that a major 
benefit of CBVA training was increased awareness about the 
vocational programs and improved relationships with the 
vocational instructors* "Going into the classrooms more" 
was repeatedly mentioned as an outstanding benefit of CBVA 
training and implementation. The site director reaffirmed 
the increase in the level of collaboration among trainees 
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stating that the resource specialists "do an excellent job 
of monitoring — they go into the classrooms two times a week 
as a routine — there's no paper involved — they talk to 
teachers and support them and don't count on the vocational 
instructors for referrals." During the meeting, numerous 
examples of work performance samples were observed and many 
others were enthusiastically described by the resource 
specialists who created them. 

In summary, data collection activities at Denver 
Career Education Center consisted of a group interview with 
the core group participants (which continued through lunch 
with the group), observation of a monthly supplemental 
services personnel meeting, and an initial and exit 
interview with the site director. Interview data focused 
mainly on the interwoven variables of system advocate 
involvement, suppc '^e work environment features, and the 
sequence of events tnat occurred during the field-test 
training and continued training. The most significant 
outcome at this site was the movement toward system-wide 
implementation of CBVA activities throughout the Denver 
Public Schools. Implementation at the Denver Career 
Education Center occurred primarily in the areas of 
placement and planning using the work sample approach for 
entry-level assessment. Other levels of implementation 
included improved communication, collaboration, and 
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awareness among the supplemental support services 
personnel^ vocational instructors, and administrators. In 
addition, the follow-up visitation served as an important 
intervention, as it resulted in the commitment from the 
assistant principal to provide monetary incentives for 
vocational instructors who participate in future CBVA 
training functions. 

Preservice site 3; University of 
Northern Colorado 

While much of the preservice follow-up data from the 
trainer at this site has been infused into the in-service 
data summary, an exit interview with the site director 
included additional information about her approach to in- 
service and preservice training, insights about 
implementation in the Denver public Schools, an unsuccess- 
ful attempt to implement CBVA in Colorado, and her future 
plans regarding CBVA. 

The site director described her involvement in 
preservice training vhich included two years of teaching an 
assessment class at the university, she listed course 
objectives as (a) an understanding of CBVA, and (b) an 
ability to actually implement it, qualifying this by 
stating that students should be able to use CBVA to at 
least meet Carl Perkins mandates in a program with little 
or no resources. 
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Her involvement with in-service training began with 
several assessments of local schools to examine their 
assessment priorities/ which led her to the Denver Career 
Education Center , where the CBVA process matched the 
assessment needs of the school. The site director stressed 
the importance of individualizing the CBVA system to the 
particular needs of the school/ "taking key features and 
saying/ this is how it works." To introduce the CBVA 
training program at the in-service site, the site director 
looked at basic concepts of student curriculum such as; 
"planning/ placement/ job try-outS/ and work samples/" but 
did not use the precise training materials. 

She emphasized that the successful implementation at 
DCEC was due/ in part/ to the fact that the Center already 
had "a lot in place" and that CBVA training should be used 
to refine and build upon what they already had. She 
repeatedly cited the outstanding qualifications of the 
system advocate who/ as supplemental services coordinator 
for Denver Public Schools/ "cleared the path" for CBVA 
training/ utilizing a newly hired team of resource 
specialists to train in and implement CBVA. In regard to 
administrative support at DCEC/ the site director did not 
see the assistant principal as truly being the 
administrator during the field-test/ but instead saw the 
system advocate in this role. She viewed his enthusiasm 
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and conunitment during the follow-up visitation as a result 
of the follow-up activity itself, described it as an 
attempt to "share in the glory" of the successful 
implementation of CBVA* 

Attempts at implementation were not as successful at 
another school where, despite strong external support from 
the site director and another university faculty member, 
key personnel "did not get it at all." The site director 
attributed this result to the lack of an effective inside 
system advocate; the individual responsible for system 
advocay was "full of knowledge" but did not have the needed 
interpersonal skills to motivate the staff. Although CBVA 
training didn't result in full implementation at this site, 
the site director saw even this partial involvement as a 
"step in the right direction," increasing awareness about 
CBVA concepts. 

The site director saw her role as an external 
supporter for in-service training as being that of an 
objective outsider. She attributed her effectiveness as a 
supporter to the fact that she was "just removed enough, 
confident," and that she "didn't take resistance from 
trainees personally." She supported training efforts by 
"addressing fears and concerns about the newness of CBVA." 

She mentioned that one area of resistance during 
training was the issue of time, explaining that it was 
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"getting kids out of class for assessment" that was 
difficult and "not development time." 

Another area emphasized during the interview with the 
preservice trainer was the support provided by her 
superiors in the Colorado Department of Education. She 
named two state level administrators who were quite 
famxliar with and supportive of her work with CBVA. The 
site director made concluding remarks that reinforced her 
great interest in CBVA and indicating that she planned to 
persue implementation at various schools, including post 
secondary schools, commenting that "calls continue to come 
in with volunteers coining forward to try CBVA." She 
described CBVA as "very positive — yet another reason to get 
into the classroom." 

In-Service Site 4; North Orange 
County ROP 

Data collection at North Orange County Regional 
Occupational Center (NOCROP) was limited to two interviews, 
one with the Director of Planning and Development, the key 
administrator who was involved with the establishment of 
the field-test program, and another with the Handicapped 
Programs and Services Specialist, who assumed the role of 
field-test coordinator. Since formative evaluation data 
collected prior to the follow-up activities indicated that 
the field-test training program had not been effective, the 
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interviews focused mainly on reflections of these 
individuals on the training program and their perceptions 
of outcomes. 

The Director of Planning and Development had not been 
directly involved with the actual training, but instead had 
reviewed and evaluated the training materials and assisted 
in the selection of the field-test participants. She felt 
that the CBVA system was excellent, the materials were 
good, but that the problem with the field-test program was 
rooted in the lack of cohesiveness among the field-test 
team members. Participants had been selected from three 
high schools within the ROP, forming three sub-teams 
consisting of representative samples of vocational 
instructors, special education teachers, and counselors 
from each school. The Director of Planning and Development 
felt that this arrangement resulted in a disjointed, 
logistically unmanageable training group which appeared to 
have detracted significantly from the effectiveness of the 
training program. 

She also noted that a recently conducted accreditation 
study had included among its findings, that the ROP was 
lacking in involvement with special education students at 
the high school level. The field-test training experience, 
she felt, had reinforced this finding. Furthermore, she 
mentioned that the position of Handicapped Programs and 
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Services Specialist, the personnel role identified as the 
system advocate for iche field-test/ did not entail much 
involvement with the high school students. Consequently ^ 
this individual rarely collaborated with the high schools* 
special needs staffs/ a situation which impaired her 
effectiveness as CBVA system advocate. 

The Director recommended thit a workable plan for 
implementing CBVA in the ROP setting be developed* She 
remarked that despite the difficulties of conducting the 
field-test at the ROP/ feedback at the district level 
suggested that the program was beneficial and that the 
training materials "gave the opportunity for staff to be 
more knowledgeable — to deal with colleagues in the area of 
assessment." 

In reference to ideas for improved training efforts in 
the future/ the Director cited more structured in-service 
training with on-*going assistance as being extremely 
helpful. She emphasized the need to ^'start small" with a 
select group/ using a "few practitioners to spread the 
concepts on to others." For field-test purposes/ she saw 
the need to "take participants all the way through the 
training as a group/ winning commitment from them from 
start to finish." 

In closing/ the Director expressed interest in 
pursuing additional CBVA training at NOCROP by stating that 
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the impact was "stronger than awareness — the field-test 
resulted in understanding of these concepts that will carry 
over into the future*" 

The second interview at NOCROP was with the 
Handicapped Programs and Services Specialist who had acted 
as system advocate during the field-test program. Her 
perception of the impact of training was that of a 
temporary improvement in communication among participants 
that "went back to the way it was" after the field-test. 
She mentioned that a chief supporter of the CBVA system^ a 
vocational instructor who was involved in the field-test 
included training in a commercial assessment reports based 
on iPTICOM and work samples. Her feelings about CBVA were 
that "it is a good idea" but that she sees the advantages 
of using more traditional "assessment center procedures." 
She cited several reasons for her view, and explained that 
the lack of distractions in an assessment center , the 
specialized training of the assessor / and the quantified r 
more objective nature of the assessment results obtained in 
the assessment center were advantages over CBVA. She 
described her current involvement with high school students 
with handicaps as being limited to "hard to place" cases ^ 
and felt that the special education staff at the high 
schools were adequately serving these students • 
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Ir summaryr findings at NOCROP indicated that outcomes 
of training were very limited* It is possible that 
training improved awareness of assessment issues and that 
the field-test will result in future training in CBVA, 
According to the key administrator involved in the field- 
test, the program brought a deficiency in the special needs 
service area to light , reinforcing the findings of an 
accreditation report. The most significant finding at 
NOCROP was the focus on the organizational difficulties of 
establishing a CBVA training program at an HOP. Also, the 
unsuitability of the position of Handicapped Programs and 
Services specialist as it existed during training, for the 
system advocate role was clearly noted, 

I 

Across-Site Data Analysis 

In a multiple-site study, greater explanatory power 
and generalizability can be achieved through an analysis of 
processes and outcomes that occur across many sites (Miles 
and Humberman 1984). The authors stated that through the 
comparision of sites, the analyst can, "establish a range 
of generality of a finding or explanation and pin down the 
conditions under which that finding will occur" (p. 151). 
The present study included a cross-site analysis of three 
in-service sites and three preservice sites; these data 
were combined for across-site analysis since within-site 
analysis revealed that interviews with preservice trainers 
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or site directors concentrated mainly on in-service 
training and implementation issues • The fourth in-service 
site, NOCROP/ was not included in the across-site analysis. 

The relatively small number of sites involved in the 
follow-up greatly simplified the across-site analysis 
process. Since it was possible to obtain a sense of 
across-site findings prior to the actual write-ups of each 
site/ the initial analysis by site was organized and 
displayed with the across-site commonalities in mind. This 
pattern of organizing and reducing data into levels of 
outcomes and implementation variables was useful for both 
within-site and across-site analysis. Extraneous data, 
those data which fell into categories outside of the two 
major areas of findings, were grouped into a third cluster, 
training recommendations. These recommendations came from 
in-service and preservice comments about effective training 
strategies based on the field-test experience. The 
identification of this third cluster for across-site 
analysis was based on a major study objective which was to 
provide data on training processes for future implementers. 

The process of across-site analysis began with a 
juxtaposition of the individual site summaries into one 
large data display. Comparisons and contrasts were more 
clearly made through the process of organizing individual 
site findings according to the various levels of outcomes 
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and by implementation variables and then displaying the 
data for all three sites in chart form. Of course, the 
individual site data summaries needed to be greatly reduced 
to make such a simplification possible. The reduction of 
data for across-site analysis was accomplished in part by 
an analysis of levels of outcome data which resulted in the 
establishment of three increasingly comprehensive outcome 
levels. Level one included improved communication, 
increased awareness, and increased collaboration; level two 
included the use of direct assessment activities for 
placement/planning, monitoring, or transitional services; 
and level three included systemwide implementation. 

The process of data reduction for across-site data 
analysis display entailed a preliminary review of data in 
terms of context, multiple source verf iciation, and 
ambiguity of responses. From here, data were ordered or 
classified, a process which Miles and Huberman referred to 
as the "transformation of narrative text into short quotes, 
summarizinng phrases, ratines, and symbols" (p. 152). For 
example, data relating to the outcome of increased 
communication at each site were assigned values of either 
high, average, low, or none based on the sense of 
generality gained through a perusal of across-site data in 
that area. These assigned values were used only as gross 
sorting tools; descriptive text was easily retrievable for 
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more fine-grained analysis. For implementation variables^ 
other rating labels, such as "excellent/ good, fair, and 
poor," to describe the system advocate; and dichotomous 
categories such as "yes and no" to denote whether or not 
continued training efforts occurred were used. The 
Unordered Meta-matrix: Levels of Implementation, Table 4, 
and the Unordered Meta-matrix: Implementation Variables, 
Table 5, show how data from each site were assembled 
coherently in one place. Through viewing the display of 
data in Tables 4 and 5, it became clear that there was a 
distinct relationship between the levels of outcomes and 
the presence or absence of the implementation variables. 
Ar Denver Career Education Center, where implementation 
occurred at all three levels, the four implementation 
variables were present as well. At Chisago Lakes High 
School, where implementation occurred most distinctly at 
level 1 but also at level 2, three of the four 
implementation variable were present. Finally, at Monroe 
High School, implementation did occur to some extent at 
levels 1 and 2, and only one of the four implementation 
variables was totally present, with a second being present, 
but withdrawn shortly after the field-test. 

This pattern is illustrated in Table 6, Site-ordered 
Predictor-outcome Matrix; Implementation Variables as 
Related to Levels of Outcomes. While this analysis process 
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Table 4. Unordered Meta-matrix: Levels of Implementation 



LI 



L2 
DA 



L3 
Mod 



Site 


Com 


Coll 


Aware 


PI 


Pn 


Mon 


Ts 


Ss 


Sys 


DCEC 


Av+ 


Av+ 


High 


High 


High 


High 


None 


None 


Yes 


CLHS 


High 


High 


High 


Av 


Av 


None 


Av 


High 


No 


MHS 


Av 


Av 


Av 


Av 


Av 


None 


Av 


Av 


NO 



Note . DCEC = Denver Career Education Center; Com = 
Communication; CLHS = Chisago Lakes High School; Aware = 
Awareness; MHS = Monroe High School; Coll = Collaboration; 
DA = Direct Assessment Activities; PI = placement; LI = 
Level 1 outcomes; Pn= Planning; L2 = Level 2 outcomes; Mon 
Services; Mod/Ss = Modifications/Support Services; Sys = 
Systemwide Implementation; 

Table 5. Unordered Meta-Matrix: Implementation Variables 



Site 



SA 



ES 



WE 



CT 



DCEC 

CLHS 
MHS 



Excellent 

Excellent 
Good 



State + 
Univ 

State 

Univ/WD* 



Positive 

Positive 
Negative 



Yes 

No 
NO 



Note. Legend for codes: DCEC= Denver Career 
Educuation Center; SA = System Advocate; CLHS = Chisago 
Lakes High School; ES = External Support; MHS = Monroe High 
School; WE = Work Environment; CT = Continued Training. 
*WD = Withdrawn. 
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represents a drastically reduced version of across-site 
data, it serves to illuminate the relationship of 
implementation variables to levels of outcomes across 
sites. Miles and Huberman (1984) advised using the site- 
ordered predictor-outcome matrix "when you want to see how 
several contributing factors function together in relation 
to different levels of a criterion measure" (p. 174). The 
use of a predictor-outcome matrix facilitates the jump from 
descriptive to explanatory analysis, testing the extent to 
which antecedents will predict certain outcomes. These 
antecedents, or implementation variables, were solidly 
grounded in the data; they surfaced unequivocally and 
consistently during the individual site data collection. 
Also, levels of outcomes were ordered on the basis of CBVA 
system objectives and verified through input from the 
project director during site analysis meetings. In 
addition, as part of the qualitative methodology, emergent 
theories were tested and re-evaluated chroughout the data 
collection process. This methodological feature resulted 
in the accumulation of data which was relevant to the 
emerging themes, contributing to the predictor-outcome 
method for across-site analysis. 

It appears form this analysis that the role of the 
system advocate, external supporter (from the state, 
university, or both) and the work environment are important 
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contributing factors to implementation. However, continued 
training, which was only present at the site which 

Table 6. Site-ordered Predictor-outcome Matrix: Implemen- 
tation Variables as Related to Levels of Outcomes 



Site Levels SA ES WE CT 

DCEC 1, 2, 3 X X XX 

CLHS 1, 2+ X X X 

MHS 1, 2 X X- 



Note. Legend for codes: DCEC = Denver Career 
Education Center; SA = System Advocate; CLHS = Chisago 
Lakes High School; ES = External Support; MHS = Monroe High 
School; WE = Work Environment; CT = Continued Training. 

demonstrated systemwide implementation, appears to be the 
critical factor in facilitating organizational change. 
Since the particular training strategies used at Denver 
Career Education Center were the only attempts at continued 
training among the three sites, it is difficult to say 
whether the systemwide changes were a direct result of 
those particular training strategies or if any continued 
training efforts would have resulted in system-wide change. 
It appeared, however, from the across-site data analysis on 
training recommendations, that suggestions for future 
training efforts from CLHS and MHS were similar to actual 
training strategies at DCEC. It seemed, then that a 



specific approach to continued training was a predictor of 
systemwide implementation. 

Alsor since the across-site analysis indicated that a 
combination of the four variables was present at DCEC; it 
is not likely that continued training alone would have 
resulted in systemwide implementation. This notion is 
supported by combined data from CLHS and MHS which clearly 
indicated that the effectiveness of the system advocate, 
the conduciveness of the work environment, and the degree 
of external support were important factors for 
implementation at the first two levels. Furthermore, data 
from DCEC indicated that the continued training efforts 
were immeshed with the other three variables. The 
successful "pyramid" approach to training clearly relied on 
the team effort of the sysyem advocate and the site 
director (external support) and the cooperative 
relationships and positive professional attitudes of key 
members of the "core" training group (working environment). 

The third cluster of across--site data grouped 
recommendations from all sites on effective training 
procedures. This cluster was not included in the unordered 
meta-matrix analysis or the predictor-outcome matrix since 
this aspect of the data seemed to be a peripheral to the 
major emergent pattern. However, since the data were 
pertinent to a major study objective, that is to provide 
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information for future trainers, these comments were 
compiled be site, compared and reduced to the following 
list which is organized in terms of how, where, and when 
training should take place, and who should be delivering 
the training: 

How: 

!• Do not use the whole system modules, but instead 
use a few "pieces" of the system for training and gradually 
build when those are mastered ♦ 

2. Individualize training to specific system needs 
(assessment priorities) ♦ 

3* Use perks, incentives, and release time to enhance 
trainee motivation* 

4* Use interagency collaboration and sharing of 
resources to coordinate the system. 

Where: 

5* Conduct training activities at least in part in 
the vocational classrooms to actually experiment with 
equipment, become familiar with materials and understand 
the program requirements. 

When: 

6. Use an on-going approach to training, with 
frequent interventions and follow-up activities, extending 
the training period to at least one to three years. 
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Who 

1. Use teacher, or local personnel who arre already 
using CBVAr to conduct the training. 

8. Use a daily monitor r technical support person^ or 
aide to assist with implementation efforts* 

Finally/ the across-site analysis revealed a common 
perception among individual site participants of the 
follow-up activity as being an important intervention* The 
follow-up visitations were reported as renewing interest 
and activity in CBVA at two sites, and stimulated 
administrative support at a third site which led to a 
commitment to provide monetary incentives for future CBVA 
training participants. All sites viewed the follow-up 
visitations as a positive activity which validated the 
field-test experience and provided recognition for 
participation in the project. 
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Chapter 5 

Summary, Conclusions/ Recommendations/ and Reflections 

Summary 

This qualitative scudy followed-up a field-test 
training program in curriculum-based vocational assessment 
involving multiple preservice and in-service sites. The 
study sought to uncover information about outcomes of the 
training program in terms of changes in the participants 
and their work environments. The primary objective of the 
study was to gain information about participants' attitudes 
and feelings about the effect of CBVA training on their 
work with students with handicaps. A qualitative 
methodology was used to obtain contextual data that would 
capture the unique conditions surrounding training at each 
site and provide idiosyncratic details about the 
implementation process. Findings of the study will be used 
to improve future training and implementation efforts. 

The original field-test training program tested a 
series of modules designed to prepare special and 
vocational education personnel in curriculum-based 
assessment activities for students with handicaps in 
vocational programs. The field-test involved on-site 
training in the systematic u£;e of CBVA activities through 
the collaborative efforts of vocational instructorS/Special 
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education teachers^ counselors, paraprofessionals, and a 
system coordinator / the vocational special education 
resource teacher. Three in-service sites / including 
Chisago Lakes High School/ Minnesota; Monroe High School # 
Georgia; North Orange County ROP/ California; and a fourth 
"spin-off" site, the Denver Career Education Center, 
Colorado participated in the three-year project, in 
addition, six preservice sites, including the University 
of Georgia, Bemidji State University, the University of 
Vermont, George Washington University, University of 
Northern Colorado, and California State University, Long 
Beach were involved in the field-test training, where 
university project staff infused the CBVA system into 
vocational and special needs courses. 

A decision to use a purposive sample of the sites 
resulted from recommendations from project staff, formative 
evaluation data, ant^ external reviewers. The follow-up 
study included four in-service site visitations: Chisago 
Lakes High School (CLHS), Monroe High School (MHS), North 
Orange County Regional Occupational Center (NOCROP), and 
the Denver Career Education Center (DCEC). Three preservice 
site directors were interviewed at those sites which 
coincided with the in-service site visits, representing a 
key informant sample of preservice sites. Twenty-eight 
individuals were involved in a series of in-specific depth 
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on-site interviews/ telephone interviews / and observations 
during the follow-up process. 

The follow-up study design was exploratory in purpose^ 
attempting to discover incidental outcomes of a "field- 
test" training program* The distinction between "field- 
test" and "actual" training brought certain methodological 
issues into focus* First/ the training materials and 
procedures were being formatively evaluated and refined 
during the field-test* Consequently, cases of full-scale 
implementation could not reasonably be expected in the 
follow-up data. Secondly/ whereas the CBVA system was 
fairly structured/ field-test training allowed for latitude 
in adapting the system to ^-he individualized needs of the 
site. Therefore/ the follow-up design clearly needed to be 
open-ended/ flexible, and focused on many levels of 
implementation. In a sense/ the follow-up study was viewed 
as a continuation of the formative evaluation process/ used 
to gain further insights into the effectiveness of CBVA and 
the implementation pirocess (Albright and Cobb 1988b). 

The follow-up study explored incidental outcomes/ 
attitude changes/ and examples of direct use of CBVA. Data 
about these impacts were organized into categories of (a) 
increased awareness about assssment issues; (b) impr >ved 
communication; (c) improved efforts to collaborate in CBVA; 
(d) the use of direct assessment for either 
placement/planning/ monitoring/ and transitional services; 



140 



132 

(e) program modifications and the use of support services 
as a result of CBVA; and (f) systemwide implementation 
involving changes in policy/ staffing, or organization 
within the work environment. These outcome categories were 
collapsed into three increasingly comprehensive levels to 
facilitate analysis. Level one outcomes included changes 
in communication/ awareness, and collaboration; level two 
outcomes included changes in assessment practices and the 
use of support services for students with handicaps; and 
level three included systemwide changes. 

On-site observations about contextual factors, 
including the role of key personnel, the school climate, 
and examples of continued CBVA training, provided 
background information to be used for the interpretation of 
data from each site. A variety of instruments, -including 
the Key Informant Tally Sheet, the Observation Summary 
Sheet, and the Document Review Sheet was used to reduce and 
display data in a meaningful way. Data were analyzed using 
a predictor-outcome matrix to determine what, if any, 
relationship existed between the various levels of 
outcomes and the observed conditions that were associated 
with the outcomes at each site. An across-site analysis, 
through which site findings were compared and contrasted, 
was used to determine the generalizability of findings 
across sites. 
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Interview responses and observations were subjected to 
several cross-checking procedures to test their validity 
and reliability. Specifically, triangulation data were 
used to weight responses on the basis of convergence of 
data about each dimension from multiple sources or through 
multiple methods. On-going analysis cycles, including 
verification of field notes with respondents and site 
trainers, and site analysis meetings with the project 
director, were used to reduce bias and increase confidence 
in the emergent theories. The data collection procedures, 
then, were increasingly productive, as they were altered 
and refined along the way. 

Conclusions 

The follow-up study resulted in a body of qualitative 
data from which clusters of training outcomes were 
identified and ordered according to levels. The first 
level of outcomes included increased comjnunication, 
collaboration, and awareness about assessment issues. 
These areas were considered basic outcomes from which more 
direct applications of CBVA training would likely develop* 
This basic outcome level occurred at all of the in-service 
sites, but was most evident at Chisago Lakes High School. 

The second level of outcomes included the use of 
components of the CBVA system which were identified 
according to the functional areas of placement/planning, 

142 




134 

monitoring^ and transitional services as they were 
organized in the CBVA system modules (Albright and Cobb 
1985). A summary of the modules is included in Appendix J. 
While CBVA was used for all of these purposes to some 
extent at each of the sites^ CBVA was most widely used for 
placement/planning purposes^ less frequently used for 
transitional services, and infrequently used for monitoring 
student performance. This pattern was true both within- 
sites and across-sites, although it was accentuated at DCEC 
where systemwide implementation entailed CBVA for 
placement/planning almost exclusively. Data from multiple 
sources indicated that this specific use of CBVA at DCEC 
was due to the individualized needs of the site rather 
than the feasibility or usefulness of this or any other 
component of the CBVA system. However, the relatively high 
concentration of CBVA activities used for placement/ 
planning at the other sites, suggested a certain amount of 
attractiveness of CBVA for that particular use among sites. 

Another outcome area clustered within the second level 
was the use of program modifications and support services 
as a result of training. This outcome was most prevalent 
at CLHS where level one outcomes were quite pronounced as 
well. It is plausible that when communication, 
collaboration, and awareness of assessment issues improved 
(level one), the direct use of CBVA activities (level two) 
would increase as well. Data from MHS were consistent with 
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this theory, indicating that examples of both level one and 
level two outcomes were sirdlarly sparse. Finally/ at 
DCECr where the third level of outcomes, systemwide 
implementation, occurred data indicated that although 
outcomes in terms of communication/ awareness, and 
collaboration (level one) were not specifically reported 
during interviews as having been a result of training, data 
did indicate that effective communication, knowledge of 
CBVA concepts and collaboration strategies were essential 
ingredients of the successful implementation process. 
This concept of increasingly comprehensive levels of 
training outcomes was clearly illustrated in the data from 
each site and across sites. Even at NOCROP, where follow- 
up data were limited, the pattern was evident; a lack of 
impact at level one (communication, collaboration, and 
awareness) seemed to prohibit further outcomes (direct use 
of CBVA) entirely. The implication of this finding is that 
training efforts which focus on achieving system objectives 
according to this pattern of acceptance and learning among 
trainees in an organizational setting are more likely to be 
successful . 

A second aspect of the follow-up study findings is 
concerned with four contextual variables which seemed to 
have had a bearing on the extent of impact across sites. 
Three site fec±ures, the role of the system advocate, the 
role of the external supporter (s) , and the nature of the 
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work environment, appeared to be directly related to the 
level of training outcomes that occurred since the field- 
test training program. A fourth, continued (post-field- 
test) training, was found to most markedly effect 
implementation; this relationship was prominently 
demonstrated in the case of DCEC. 

Across-site analysis revealed that the combined 
elements of effective system advocate leadership, active 
support from a university or the state, and a positive, 
progressive work environment contributed to a setting that 
was most conducive to implementing the CBVA system. The 
follow-up data revealed that the presence of all of these 
factors was related to strong incidence of level one and 
two outcomes (CLHS); the partial presence of these factors 
was related to moderate incidence of level one and two 
outcomes (MRS) ; and the absence of these factors was relate 
to the near absence of all levels of outcomes (NOCROP) . 
Furthermore, when these three variables (system advocate, 
external supporter, and work environment) were present 
along with the fourth (post-field-test training efforts) 
all three levels of outcomes occurred. Most significantly, 
the presence of this fourth variable at DCEC resulted in 
full-scale implementation of the CBVA system at the Center 
which eventually spread throughout the Denver public 
schools. These findings suggest that consideration of 
these implementation variables would be benefical when 
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selecting a site for future CBVA training and in designing 
the training program^ 

Aside from the focus on analyzing the follow-up data 
for outcome levels and implementation variables, a set of 
key training guidelines was gleaned from the strategies 
described at DCEC and recommendations from field-site 
trainers and participants about the dynamics of system 
change as they perceived them during and following the CBVA 
field-test training program. In essence, these suggestions 
emphasized (a) a piece-by-piece building approach to 
training, both in terms of content and the tr^ ning group; 
(b) an intensive, extended training perrod involving 
frequent interventions and follow-up activities; (c) a 
training staff consisting of teachers or local personnel 
and technical support aides; (d) an in-class training 
component to allow for hands-on exposure to the vocational 
program; and (e) an individualized training approach based 
on assessment priorities at the particular school. 

Interestingly, the impact of the follow-up study 
itself was noted consistently across in-" rrvice sites. The 
follow-up was viewed as an important intervention; data 
indicated that interest in CBVA was renewed as a direct 
result of the visitation, and in some cases, commitments 
for future involvement with CBVA were made. 

In summary, several conclusions were drawn from this 
study about the impact of the field-test training on 
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^••rticipants and their work environment. Training outcomes 
had occurred at three levels, encompassing basic effects 
such as improved communication and improved awareness, more 
defined impacts such as the use of CBVA practices, and 
comprehensive changes in the organizational setting. A 
theory of incremental implementation emerged from the data, 
as a way of explaining the pattern of outcome levels that 
occurred both within sites and across sites. A high 
concentration of data indicating the use of CBVA for 
placement/planning purposes was observed and was 
interpreted as a function of assessment priorities at the 
sites. Recommendations for training and implementation 
efforts were pulled from the data and analyzed, resulting 
in a set of guidelines for future trainers about such items 
as how, when, where, and by whom training would most 
effectively be conducted. Last, the impact of the follow- 
up study itself was reported to be an important 
intervention at each of the sites. 

Recommendations 
Based on the study findings, recommendations are 
focused mainly on future CBVA training and implementation 
efforts. Findings of this follow-up study indicated that 
the field-test training program produced a wide range of 
impacts on participants and their work environments. It 
appeared that these changes evolved in a similar pattern 
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across sites and that key contextual factors contributed to 
the extent of impact as well. Future training efforts 
which start by conducting an initial site assessment of (a) 
personnel/ (b) organization, and (c) current assessment 
practices could use this information to establish the 
conduciveness of the site for implementation and to provide 
a basis upon which a training plan could be developed. The 
initial site assessment would include an analysis of 
implementation variables, as they were identified in the 
follow-up study, to identify "strengths and weaknesses" of 
the site as they relate to the implementation process. If, 
for example, no suitable system advocate existed at a site, 
an appropriate individual might be coached prior to 
training as to the significance of his or her role. In 
short, a missing key variable, the role of the system 
advocate, could be compensated for early in the training 
process, in an effort to maximize the existing conditions 
and facilitate the implementation process. 

The use of an "incremental" approach to training seems 
to be an effective way to move toward full implementation 
of the CBVA system in an organization. Through this 
approach, training would begin at the basic level of 
communication and awareness and continue to build on this 
foundation with efforts to effect change in direct 
assessment practices and program modifications. Specific 
functional assessment areas, such as placement/planning, 
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monitoring, or transitional services, could be covered 
incrementally as well. As each of these levels of change 
were achieved, organizational change would become more 
accessible. 

These two recommendations, for a pre-training site 
assessment and for an incremental implementation approach, 
are not to be treated as two entirely separate issues. The 
site assessment, uncovering information about the work 
environment and current assessment practices, can be used 
as a tool for gauging the need for concentration of 
training ot each implementation level. For instance, when 
system advocat-^ leadership, external support, and work 
environment are all supportive of implementation, the need 
to train at the basic communication level would diminish^ 
Conversely, when these variables are lacking at a site, 
greater attention to issues such as interdisciplinary 
communication and collaboration or to the concept of 
curriculum-based assessment might be helpful before moving 
into specific training on particular system components* 

Mention of the fourth implementation variable, 
continued training, has been reserved for separate 
discussion, since the context of "recommendations for 
future training" changes the meaning of this variable as it 
related to the field-test training program. In terms of 
future implementation efforts, continued training would oe 
a given, not an incidental outcome of field-test training* 
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However/ the significance of the continuity of training 
efforts/ of systematic training and retraining cycles/ as 
they occurred at Denver Career Education Center/ is noted 
and should be incorporated into the implementation process 
at other sites. The point is that a well-organized 
procedure is easily repeated; the simple/ yet useful •'try- 
out" modules used as a basis for training at DCEC 
represented a process for implementing CBVA that was 
quickly integrated into the organizational setting. 

Other "training guidelines" as they emerged from the 
follow-up study/ deserve attention in that they were 
constructed from practioners and trainers/ who had the 
unique status of having beer "field-test participants." In 
this capacity/ these informants concurrently experienced 
and evaluated the training process and for whatever 
complications resulted from this dual focuS/ a benefit 
appears to have been the insight they shared about \ .lat 
worked/ didn't work/ and might have worked during and after 
the field-test. 

The recommendations of participants at each of the 
siteS/ regardless of the outcomes that occurred at their 
site were surprisingly consistent; in fact/ many of these 
"training guidelines" were training techniques or 
contextual factors highlighted at Denver Career Center 
where implementation was successful. Specifically/ the 
emphasis (a) the use of local personnel/ or "teachers who 
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have used CBVA** to conduct training, (b) the individualized 
tailoring of the CBVA system to site needs and priorities, 
(c) the "piece by piece" building approach to training 
content and group, and (d) the use of on-going, on-site 
training with extensive technical support and interventions 
were major elements of the proposed strategies for 
effective training and implementation. Interestingly, two 
of these reconunendations, for individualized training 
based on a site assessment and for an incremental approach 
to training, parallel the overall study recommendations. 
This consistency enhances the credibility of both the 
informants' perceptions and the study findings, it is 
seems rather clear, then, that these guidelines would be 
valuable to the future CBVA trainer. 

The importance of the follow-up study cannot be 
overemphasized; the admission of the impact of the follow- 
up visitations by participants and the observed effects 
strongly indicated the follow-up activity was a critical 
phase of the training process. Trainers should develop an 
effective follow-up design as both a means of stimulating 
outcomes and for collecting formative evaluation data. 

Reflections 

The present study, which used qualitative methods to 
follow-up a field-test training program, brought to light a 
true sense of the difficulties and rewards of this type of 
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study. Several realizations, about the strengths and 
weaknesses of the study, are discussed here to contribute 
to the continued, and much needed efforts in qualitative 
research. 

One of the strengths of qualitative research is the 
depth of data that can be obtained; the rich, contextual 
data collected through interviews and on-site observations 
allow for a holistic interpretation of behaviors, events, 
and responses. The qualitative researcher is struck by the 
vividness of his or her recollections about conversations 
and personalitites as he or she reviews and analyzes data. 
Still, the "undeniability" of the qualitative study is 
probably questioned most by the researcher, who is 
uncomfortable with the heavy dependence on judgmental, 
discriminatory, and intuitive thinking that is an integral 
part of this methodology. 

Fortunately, however, the knowledge base on 
qualitative data analysis is expanding; the work done by 
Miles and Huberman (1984) is an excellent example of the 
extensive guidelines which exist for using qualitative 
methods in educational research. The intent of these 
authors, was to find qualitative data analysis methods that 
were "practical, communicable, and non-self deluding — in 
short, scientific, in the best sense of the word." (p»15) 
Since this study used personal interviews as the primary 
data collection instrument, most of the conclusions are 
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based on what Van Maanen (1979) refers to as 
"presentational* data. In his discussion of field research 
methods, the author differentiates between "operational" 
(observed) data, which are often spontaneous conversation 
and activities, and "presentational" data, which concern 
the appearances the informants wish to maintain or enhance 
through the reporting of abstract, idealized data (p. 42). 

The mistaking of presentational for operational data 
is one of the pitfalls the qualitative researcher must 
avoid. Van Maanen offers a solution: "Inference and trust 
are central matters here and therefore evaluating the 
believability of what one hears and sees is critical . . 
(p. 44). While this "evaluation of believability" seems 
like a highly subjective task, in actuality, it is a labor- 
intensive procedure that entails the same objective 
measures used in quantitative methods including counting, 
verifying, and testing procedures. 

While there will always be some degree of doubt about 
the believability of "talk-based" data, the researcher can 
use a variety of criteria through which the strength of 
presentational data can be determined. For the present 
study, the researcher weighed participant responses 
according to criteria as defined by Miles and Huberman 
(1984) in an effort to reduce the risk of being mislead by 
presentational data. Specifically, these authors provided 
a list of circumstances that strengthen data: (a) when data 
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are reported firsthand, (b) when the fieldworker is 
trusted/ (c) when data are collected in an offical or 
formal setting, (d) when data are volunteered to the 
fieldworker, and (e) when the respondent is alone with the 
fieldworker* 

A second area of concern for the researcher was the 
reliability of field note interpretation. Although the 
present study did include getting immediate feedback from 
participants on interview data, an effort to obtain 
corroboration from participants on overall site findings 
was not done in this study. The use of participant 
reviewers of case study reports as illustrated by Albright 
(1979) would have increased confidence in study findings as 
well as served as an additional source of follow-up data. 

To further enhance reliability of this qualitative 
study, it is noted that raw data, including field notes, 
documents, and other study records are available for those 
interested in the methodology used in this type of study. 
Miles and Huberman (1984) used Cuba's term — "audit trail" 
(p. 244) to describe the documentation of data and 
procedures from start to finish as a means of providing a 
retraceable, confirmable process of analysis. Appendices A 
through H contain samples of data as they were transformed 
through various stages of reduction and analysis in this 
study to supplement records for auditing purposes. 



In conclusion, the qualitative analysis endeavor is 
challenging* Ther b seems to be no simple way to approach 
this type of data. However, as research efforts using 
qualitative methodology continue and these studies are 
documented and shared, this type of study can be more 
effectively used to uncover the complexities of 
organizational change in education. 
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Appendix A 
Interview L^-ionary Sheet 
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SITE, A^iy^ -Cui45^ / . \ \/A 

INTERVIEW SUMMARY SHEET 



Outcome 



Code 



Pi/ . ^ 



7 JV- 



Context 

<> 0 



Code 



t1 'S K-f^/ff 



0 



'^'Tju-v^ /t/.,ro i)c^, hirfo. ^ 7 v.'^ 
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KEY INFORMANT CHECKLIST 



INFORMANT 



Ul 



S3f ( a-^-^^ ) 



IDENTIFIED PERSON 



REHAIUCS 



(^4^ -TltK c^^^ Lk/v) 
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OBSERVATION SUMMARY SHEET 



SITE: ^ 
MAJOR OBSERVATIONS 

"^>l^cd> ^A*. . r^UcJJ-i^ 



^^'^^ ^*^-/f^'^''> 



REMARKS 
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Appendix D 
Site Analysis 
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Sice Analysis 



Site ; CLHS 
Significant Outcomes: 

Instructors, spec. ed. teachers, and VSERT are all actively involved in using 
CBVA procedures to assist special needs students in the areas of: 

- conaminicating program options (parent foror slide interest battery) 

- planning (lEP involvement) 
" monitoring 

- transitional services (Diane) 

* slide interest battery - VJERT 

* support math program for V.E. programs 

* hiring 0^ Job coach for supportive employment 

* numerous examples of perforaance samples - entry-level assessment ii.struments 

tools used for on-going assessment - monitoring 

* plans to have a career night to inform parents/students about program 
options — use slide interest battery as visual presentation 
exploratory vocational curriculum-for s.n. 2 week segments of a variety 
of vocational programs 



Emerging Themes: 



1. 



2. 



The Involvement of a highly valued, influential individual as system 
advocate/coordinator of training is essential for implementation. 

The follow-up activities a: low an opportunity for project staff /participants 
to rethink Importance of CBVA and to renew interest i»nd productivity. 

School environment positive, progressive—personnel viewed themselves 
as a winning team" - spirit of cooperation, commitment, professional- 
ism — sense of pride were apparent. 

(Len) High level of state support for vocational education is advantageous 
tor schools implementing an innovative program, (for resources and diffusing 
and disseminating approaches and products. 
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Interview Schedule 1: Trainers 
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Interview Schedule 1: Trainers 

Emphasis: a) Data regarding impact of pre-service data and 
b) Directions for follow-up of in-service training. 

1. You were the pre-service trainer at site X. How would 
you describe that experience? Generally positive? 
Negative? 

2. Why did you chose this site? 

3. Do you feel that pre-service participants had a good 
grasp on the philosophy, concepts, and techniques 
associated with CBVA training? Prior to and/or after 
the training program? 

4- Since completion of the training program have you 
received any information regarding changes at the 
site(s) in terns of assessment practices and policies? 
Do you see these as a result of training? 

5, You were involved with in-service and pre-service 

training. To what extent are you presently involved 
in CBVA training activities? Related course? 
Consulting? Workshops? 

5. You were also involved to some extent in the in- 
service training at site X. Could you describe that 
involvement? 
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7. Have you received any information regarding outcomes 
of in-service training at X site? 

8. Is there a particular individual or group of 
individuals from whom you feel I could get information 
about impacts of in-service training? 
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Interview schedule 2: System Advocate (VSERT) 
Emphasis: a) Data regarding identification of key 
implenenter and b) System-wide impactsof in-service 
training ♦ 

!• How would you describe your role here at (site X)? 

2. Would you describe yourself as the lead person in the 
field-test training? Did you function more as a 
coordinator, or a system advocate? If not, whom do 
you see as the lead person within your school? 

3. How would you describe the field-test training? 
Generally negative or positive? 

4* Do you feel you have a good understanding of the basic 
concepts of CBVA? characteristics of CBVA? 

5- Do you feel the other trainees understand CBVA 
processes. CBVA concepts? 

6. Has you notion of assessment changed as a result of 
CBVA training? How? 

?• Has your function as changed as a result 

of training? How? 
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9. 



10. 



11. 



12. 



Have you observed any changes in the assessment 
practices of your colleagues since training? Please 
identity them by personnel role. 

(If yes to above) Do these changes relate to 
placement, planning, monitoring, or transitional 
services? 

Have you been involved in any further training related 
to CBVA? 

What would you say was the greatest benefit of CBVA 
training? for your school? for you? 

What would you say was the most outstanding problem 
with the training program? 
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Inter rview Schedule 3: Administrators 
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Interview Schedule 3: Admin istxators 

Emphasis: a) Data regarding administrative support of in- 
service training, b) system-wide impacts of training. 

1. How would you describe your involvei nt in the in- 
service training? 

2. How would you describe current assessment practices at 
your school? For placement and planning? Monitoring? 
Transitional services? 

3. Have any cf these practices changed as a result of 
CBVA training? 

4. Have you observed any changes in policy, resource 
commitments, staffing at your school which you feel 
are a result of CBVA training? 

5. What is you sense of the reaction of staff members who 
participated in the CBVA training program to the 
training? 

6. Have you observed any change in attitude among staff 
regarding assessment practices since training? 

7. Whom would you recommend as a contact person for 
additional follow-up information? 
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Interview Schedule 4: Vocational Instructors, 
Special Education Personnel, Guidance Counselors 
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Interview Schedule 4: Vocational Instructors, speciax Kducation 
personnel. Guidance Counselors 

Emphasis: a) Data regarding impacts of training on their 
assessment practices with students, b) collaboration with 
colleagues, or c)awareness of support services, etc.)- 

Placement 

1. Are you involved in any assessment procedures related 
to the placement of special needs students in your 
program? 

Do you take a le'id role in that assessment activity? 
collaborative? minor? 

Has this function changed as a result of CBVA 
training? 

Environ ment Assessment 

2. Do you assess your classroom environment in terras of 
equipment, materials/ teaching methods and style with 
regard to special needs students? Do you make 
adjustments to a program to accomodate special needs 
students in your class? 

3. Do you communicate this information to counselors, 
special education personnel, students, or parents? 
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4. Are you ever approached by counselors, vocational 
instructors, special education personnel, students or 
parents about the classroom environment with regard to 
a special needs student? 

5. Have any of these assessment activites related to 
student placement and planning changed as a result of 
CEVA training? 

Planning 

6. How do you plan for a special needs stu^'ent's 
program? With whom do you collaborate? Do you 
perceive these activities as being a result of CBVA 
training? 

7. Do you use support service to help special needs 
students succeed in your program? 

8. How do you select these services? Do you receive 
assistdi ce in selecting these servcices? Do you 
perceive your use of support services as having 
changed since CBVA training? 

Monitoring 

9. Do you monitor special needs students tc determine if 
and when they meet objectives and if and when they 
need support services? What techniques do you use? 
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How often do you monitor these students' progress? 

10. What kind of interventions do you make when they're 
needed? 

Do you typically communicate this information to 
parents, students, counselors, etc.? 

11. Do you see your monitoring techniques as having 
changed as a result of CBVA training? 

Transiti onal Services 

12. How do your assess the level of proficiency of special 
needs students as they complete your program? 

Does the student take part in the evaluation of his or 
her progress? 

Do you meet with other staff members or perents to 
discuss this evaluation? 

Do you see these evaluation procedures as having 
changed as a result of CBVA training? 

13. Do you help p^an for a special needs students 's 
activities subsequent to his or her completion of your 
program? 

Do you assist him or her with reviewing available 
options? 
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Do you percieve your involvement in a student's 
transitional 

plan as having changed as a result of CBVA training? 
Professional DAVPlnpmenr 

14. Are you involved in any professional development 
activites related to the CBVA training? (Courses, 
workshops, seminars, etc.) 

General 

15. What would you describe as the most significant 
outcome of CBVA training? 
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Data Susunary Sheet 



Site: MN - Chisago Lakes High School 

Outcomes 

Communication 

1. There is improved communication among trainees. 

better relations between VI, and SSE's. 

better understanding of each other's jobs. 

better understanding of various vocational programs. 

better relations between VC'r and vi's and SSE's. 

better understanding (VI's and SSE's) of the 

registration process. 



Comments : 

1. "It brought us together in one place, usually there is 
no tiir for this," 

2. "Communication opened up through trainint." 

3. "Training really brought the vocational instructors 
together. " 

4. "^ne most outstanding benefit was the awareness of 
what each other did and that it brought the staff 
together." 

5. "Training was excellent, though — getting teachers 
together and finding out what the Special Education 
people do — was great!" 

6. "There has always been a general feeling among tsachers 

that Special Education teachers have a low student 
ratio — an easier job — and there has been some 
apprehension about this. CBVA training has alleviated 
some of this." 

7. "The comraderie between vocational instructors has 
improved'-- we're more aware of each other's programs • 
It brought people together as friends." 

8. "It was critical that members got together and 
diS' ssed — this was a vehicle for communication." 
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9^ "It is important that we're all coming from the same 
angle*" (Mentioned in discussion of benefits of CBVA 
as a coiDiaunication tool*) 

10. "Now they <VSERT and VI 's> know what each other 
wants — they know what expectations are now." 

11, "Now we have a more m-depth understanding of what is 
done in different vocational education programs." 

"It brought teachers together — was an official 
connection — this never happened before." 

13. "This IS really VSERT's job, but now we're more open 
to her ideas — more flexible, because we know where 
she's coming from." 

Summary : 

13 responses indicating that overall benefit of CBVA 
training was improved communication among staff 
members : 

SSE ~ 1 (4x) =4 
VI - 1,1,1,1, = 4 
VSERT - 1 (2X) = 2 
VC - 1,1 (2X)G, = 3 

frwg^ TI-eness 

2 There is an increased awareness of assessment issues 
and concepts as they relate to special needs students 
among trainees. 

more aware of current (pre-CBVA) assessment practices 
and how to improve upon them. 

more aware of assessment areas that need work, i.e. 
transitional services, placement, planning, and 
monitoring 

changed perception of assessment - to identify 
weaknesses, not as a "weeding tool". 

"consiousness-raising" about special needs students 



Comments: 

1, "I'm more aware of the assessment procedures being 
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2* 
3* 

4. 

5. 

8. 

9* 

10* 

II* 

12. 

13- 

14. 
15. 

16. 



used by counselors. and am better informed on how to 
use them with my <special education> students." 

"We need to do more work in the area of transitional 
services for special needs stuaents— we see this as an 
ares lacking as a result of CBVA training." 

"We (VI's)s would like to see more help in the area of 
screening — we need help with looking at curriculum and 
making adjustments for special needs students — with 
the ecological assessments." 

"There's been a change in awareness of each other's 
programs, of special education tests, functions— and I 
think more about assessment of special needs people." 



"I use assessment, not as a weeding tool, but for 
identifying weaknesses." 

"We realized we need to do more work on placement, and 
career planning." 

"My perception has definitely changed — I'm so much 
more aware of things to change." 

"I find time because we must meet their <special 
education student's> needs to help them survive". 

"My monitoring methods are informal — I need to work on 
these." 

"They all realized the importance of CB^. . for special 
needs kids." 

"Training more vividly pointed out to instructors the 
student needs." 

"We need to look more at transitional service options 
after we've done all we can <for a special needs 
student>. 

"Training has had a consciousness-raising effect on 
all of us." 

"We pay more attention to prerequisites." 

"Now we look more at work samples as an assessment 
tool . " 

"I'm now more aware of what's expected of regular 
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17, "I now have a better handle on when there's a problem 
with a special need student," 

Siimmary: 

17 responses indicating an increased awareness about 
assessment and special needs students 

SSE - 1 (4x) = 4 

VI - 1, 1 (5X) = 6 

VSERT - 1 (3X) = 3 

VC - 1 (2X), 1 (2X) = 4 

Collaboration 

3. Trainees are collaborating to conduct CBVA 

activities and are utilizing the results to ensure 
success for special needs students. 

Vocational instructors are providxng special education 
teachers with information about their programs to be 
used for placement and planning assessment activities* 

Vocational counselor is visiting classrooms to become 
more familiar with vocational programs to aid in 
assessment activities for placement. 

Vocational counselors collaborate with VSERT for 
registration of special needs students. 

Vocational instructors ajre collaborating with VSERT to 
provide support services for special needs studentr?. 

Vocational instructors are taking part in the lEP 
process • 

Vocational instructors are collaborating with VSERT to 
create assessment tools and make program 
modifications . 



Comments : 

1. SSE now collaborates with horticulture VI to assist in 
developing science curriculum for his special needs 
students . 

2. VI explained to other VI in a joint interview that 
when he needed help with a student^ he would go to 
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VSERT and that she would help him, stressing the need 
to initiate the contact • 

3. VI now collaborates with VSERT and SSE and is "more 
likely to catch problems earlier by discussing 
deficiencies with them." 

4. VI explained that she was more apt to use special 
education services as a result of training — now she 
"demands it." 

5. VI explained that now "she finds a way to get help — 
before I got frustrated and either lost the student or 
<the jtudent> had behavioral problems." 

6. VI describrd case of special needs student who became 
quite successful in his program and attributes her 
progress to his collaboration with VSERT which 
resulted in arranging an extra hour per iek for the 
student to keep up. 

7. VI now collaborates with VSERT and SSE "ro find a way 
for them <special needs students> to pass — otherwise 
they would have failed." 

8. VI explained how "after first week, I go and get 
needed support from SSE." 

9. VI statea that she was "almost always involved in the 
lEP process — and if it's not possible for me to 
attend, I provide written input." 

10. VI described an assessment tool she and VSERT created 
which is now used for placement in her program. 

11. VC's explained how the registration process had 
changed as a result of training and that now a list of 
special needs students entering the schooi would be 
sent immediately to the VSERT for placements 

12. VI works on lEPs now and helps to identify realistic 
goals for special needs students in her program. 

13. VC stated that he qezs into the classroom more now and 
sees how important this is for assisting in the 
placement process. 

14. VI described how he collabored with VSERT and SSE to 
revise the entry-level skill assessment tool he 
developed in automotive program. 
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Summary: 



14 responses indicated that staff members were 
collaborating to provide assessment services for 
special needs students as a result of training. 

II ^ ^' ^' ^ ^2^)' ^ (2x), 1 (?x) = 11 
VC - 1 (2x) = 2 
SSE -1=1 

Direct. Assessmen t Activiti<^ <^ 

4. Trainees are using assessment activities which are 
directly integrated into the curriculum of the 
student's vocational program including, 

direct assessment activities during placement and planning; 



Comments : 

1. "I learned what the Career Assessment Inventory was 
as a result of CBVA and learned how to adapt it to use 
with my special needs students. I go through it with 
them one by one and although it's time consuming, it's 
very helpful." ^ 

2. "I developed a form for communication with the parents 
which I use at lEP meetings. I've gotten some very 
interesting inf ormati jn from parents which helps me 
with planning." 

3. A vocational instructor developed a general 
automotive background assessment tool to be used to 
asses entry-level skill and to facilitate planning of 
student's program. 

4. A vocational instructor uses a perspective drawing 
assignment as an entry-level assessment tool to 
facilitate planning of a student's program. 

5. A vocational instructor uses the tools she developed 
during training as assessment tools for planning 
students' programs and finds that somo of the 
materials she had been previously using can now 
effectively be used as assessment tools for planning 
purposes. ^ 

6. Vocational instructor uses equipment ID assessment 
instrument for assessing skill level at entry. 
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1 . VSERT developed slide interest battery to assess 

student interest in vocational programs in the 10th 
grade and uses data for placement uL special needs 
students. 

8. SSE developed and teaches a third alternative 

vocational math support class as a prerequisite for 
special needs students going into vocational classes 
which require a certain level of math skill. The 
program was developed through collaboration with all 
vocational instructors and the VSERT by assessing 
vocational programs for math skill reauirements for 
entry. 



and direct assessment activites during transition from 
program into employment or alternative: 

comments: 

9. "Training helped me to formulate better questions for 
social service agencies, like the Department of 
Vocational Services and I found out more about the Day 
Activity Center (sheltered workshop). 

Summary - 9 responses indicating trainees were using direct 
assessment activities during placement, planning, and 
transition into employment or an alternative. 

VI - 1, 1, 1 (2x) = 4 
SSE - 1 (3x) , 1=4 
VSERT -1=1 

ErQqram Modifications an d Support Services 

5. Trainees have used CBVA data to make program 

modifications and to provide support services to 
ensure the success of special needs student in their 
programs . 

Vocational instructors have developed a number of 
strategies for adapting curriculum and instruction to 
support special needs students in their programs. 

Vocational instructors are using aides, peer tutoring, 
and special education services to ensure the success 
of special needr students in their programs. 

Comments : 
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!• Special education teacher uses two aides for 

monitoring student progress and for providing constant 
support in the vocational classrocws, 

2. Vocational instructor broke task lists for his program 
into finer detail, slowed time frame, and uses special 
services to tailor program to special needs students 
and finds that they're more successful. 

3. Vocational instructor more clearly defined semesters 
of his program to accomodate special needs students. 

4. Vocational instructor uses aides for mentoring 
progress of special needs students in her program. 

5. Vocational instructor uses peer tutoring for special 
needs students in her program • 

6. Vocational instructor useb labs, demos, aides, 
worksheets, individual instruction, and reads tests 
aloud as support services for her special needs 
students. 

7. Vocational instructor uses computer tutorial for 
extra keyboarding practice for special needs students 
in his program. 

8. Vocational instructor says aides are very important 
in ensuring success for his special needs students. 

9. Vocational instructor uses an aide one to two days a 
week as support for her special needs students. 

10. Vocational instructor saw need in curriculum to ''add 
steps between A and B" for special needs students in 
her program. 

11. Vocational instructor developed additional worksheets 
in skill development for her special needs students. 

Summary: ll responses indicated that trainees wer^ 

adapting their curriculum and instruction and using 
support services to ensure the success of special 
needs students in their programs. 

VI - 1 (2x), 1 (2x), 1, 1 (2x), 1 (3x) = 10 
SSE -1=1 

Satisfaction wit h Training 

6. Trainees perceived CBVA as being a viable system of 
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assessBent and noted several benefits of the training. 
Comments: 

1. Academic instructor (former vocational instructor) 
felt the system had great possibilities: it avoided 
duplication of efforts, will increase efficiency after 
start-up time, will allow teachers to keep closer tabs 
on students, allow for more interventions — quicker, 
more vocational instructor involvement in the lEP 
process, and more involvement of parents. 

2. Academic instructor found training interesting and 
stated that although he liked the discussion foxnnat, 
he felt that others probably would have benefited more 
from a step-by-step delineation of modules. He also 
recommended that the system include mechanisms for 
secondary /post-secondary linkage. 



3. VSERT felt that CBVA is "the way to go—because it 

makes use of the resources and people we have." Also 
commented that CBVA is a "common ground", "much needed 
tool" and a "great start for charges". After 15 years 
in the field, she said that she had seen the need for 
CBVA from the beginning. 

Summary: 2 respondents specifically commented on the 
positive aspects o. the CBVA system and' training. 



Trainees were skeptical of or had ambivalent feelings 
about CBVA training. 



Vocational instructor said he had been skeptical of 
training He "felt his class was not conducive to 
CBVA becc use it was not really an entry-level class 
and it required a lot of writing. Ks explained that 
this was true because his program required "intrinsic 
and extrinisic motivational skills" and a strong math 
background. Also, he explained, it was toe difficult 
to assist students on an individual basis because he 
had such a large class. ^ He felt available tutorial 
services were, not reliable and that special education 
services were loosely and he had not had much success 
in using them. He commented that "you could have all 



AI/VI - 1 
VS^RT - 1 



Comments : 
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the training in the world, but no time to do it." His 
ending coiruaent was, however, that "training was 
excellent, though, getting teachers together and 
finding out what Special education people do was 
great! " 

2* A vocational instructor commented that training was 
greaw, but that CBVA was not feasible, tiiat she had 
had her doubts all the way. She commented on how 
there were always "studies, studies, studies, and no 
time to do these things." Her final comment was that 
the only way it could work would be with aides and 
release time to work on it. 

Summary: 2 respondents made comments that expressed 
skeptical or ambivalent feelings about training and the 
feasibility of CBVA training. 

VI - 1, 1=2 

Role of Administration 

8. There was little or no involvement of adwinistration 
in training process or implementation. 

Coinments : 

"We were in between superintendents during t'.ie CBVA 
training and there was little involvement from 
administration . " 

"We were switching superintendents during training and they 
were not involved. The administration likes to try new 
things for P.R. reasons but there's no real support — no 
release time." 

"The administration is supportive if thexe is a need — 
they're change oriented." 

Summary: 3 responses indicated that adiainistration had 
liitle or no involvement in the training process or 
implementation except to endorses the decision to 
participate as a field-test site. 

VI - 2 
SSE - 1 

Professional Development 

9. Participants felt that training had impacted their 
professional development by eithnr expanding their 
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role or stimulating them to pursue further 
professional development in special education or 
assessment of special needs students. 

Comments : 

1. VSERT sees her role as expanding; she sees herself 
doing more work in transitional services, working with 
the community agencies. She's presently working on 
an employment package v/ith employers involving 
financial incentives for her work experience class. 

2. Vocational instructor sees herself possibly taking 
more courses in vocational education or special 
education towards a masters degree. She now sees the 
need for special education support in curriculum. 

Summary: 2 responses indicated th::t trainees felt that 
training impacted their prof essicv<al development by either 
expanding their professional role or as a stimulus to 
pursue further education in the area of assessment of 
special needs students. 

VSERT -1=1 
VI - 1 = 1 



Observations: 
School rlimat;^ 

1. Personnel at this site demonstrated a positve and 
cooperative attitude during follow-up activites. 

All trainees attended a reception on Monday morning 
for follow-up orientation. 

During tour of facilities teachers were eager to talk 
about their programs and enthusiastically des.cribed 
how they were using CBVA in their programs. 

Office personnel offered their assistance <ind support 
during follow-up activities. 

VSERT had scheduled all trainees to participate and be 
interviewed for follow-up as a response to my request 
t.0 include anyone whom she thought could provide 
useful follow-up data. All participants were 




ERIC 



available for interviews according to the schedule she 
prepared* 



2. The site demonstrated pride in their staff and 
facilities . 

Comments: 

"We're pleased with the staff and facilities — everyone 
is very cooperative." 

"The school was picked by the state as a school of 
excellence." 

"This is an impressive group of vocational 
instructors — we're unique in thir respect." 

"I feel that this school is progressive and despite other 
offers, I choose to stay here." 

"The schrol is involved in a lot of innovative state 
programs — we're a model school." 

The business program had received highest local award — the 
"SPIRIT" cwara and the walls of the businers program 
office were covered with awards and letters of recognition. 

The VSERT kept a scrap book for the work experience program 
which documenting years of community events / newspaper 
'lippings, and student projects and accomplishments. 

System Advocate 

3. The system advocate, the VSERT, was a highly valued 
staff member. 

- She had gained the support of all staff members for the 
follow-up activities. 

- She was repeatedly mentioned during interviews as 
recorded on the Key Informant Tally, and praised for her 
outstanding qualities. 

"She knows her 30b well." 
"She's very organized." 

"She's good about providing background on special needs 
students I have in my program." 
"Her heart is there." 

"She has the background, and knowledge." 
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••She's a mover and shaker," 

"She wears easily — she has a good personality." 
"She can talk you into something nicely — not 
aggressively. " 

Comments came from vocational instructors, special 
education personnel, and counsjlo^s, and administrator. 
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TrQin«f's Manual 

Cumculum-Bosed Vocational Assessment 
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ABSTRACT 
Module 1 

Establiflhing a Curriciilttni*Baged 
Vocational Assegsment Process 

GOAL This module has two interzelated goals: 

1 . To present a collaborative team process for 
establishing curnculum*based assess- 
ment practices in local vocational educa- 
tion programs. 

2. To increase participant understanding of 
the full range of vocational education and 
special services available to students with 
handicaps in the local district. 



CONTENT Module 1 consists of two parts. Part I provides 

SUMMARY a series of steps and ccosiderations for local 

team members to use in establishing a CBVA 
process. A msQor outcome of this activity is the 
development of a local action plan for expand- 
ing and improving assessment services. 

Part II is concerned with procedures for ensur- 
ing that parents, students and educators are 
knowledgeable of the programs and services 
available through vocational education in the 
school district. Strategies for increasingparent. 
student and educator understanding of these 
programs and services are provided 



KEY ASSESSMENT 1. To what extent are students with hand- 
QUESTIONS icaps» their parents and educators aware 

of and knowledgeable about vocational 
education and special services available iii 
our district? 

2. What are the vocational assessment prac- 
tices and needs within our district? 

3. What changes will be necessar>'ia order to 
successfully establish a CBVA process 
within our school district? 

Note. From L. Albright, 1988c, Assessment of 
students with handicaps in vocational education; A 
curriculum-based approach , Arlington: American 
Vocational Association, p. 15. 
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Trcrfn^f's Manual 

Cunlculum-Sased vocotfonal Assessment 
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ABSTRACT 
Module 2 

Understanding Carriculuin*Based Vocational Assessment: 
Porpofles and Characteriatica 



GOAL 



To increase partidpant knowledge of: 

1. The central purposes and features of a 
CBVA process; and 

2. The relationship between a CBVA process 
and the content of this training program. 



CONTENT 
SUMMARY 



The first section of the module is a descnption 
of the key elements of CBVA. The second sec- 
tion illustrates how the training program is or* 
ganized around the miyor purposes of a local 
CBVA effort. 



KEY ASSESSMENT 
QUESTIONS 



1. What is CBVA? 

2. How is the content of the training 
program organized? 



Note. From L. Albright, 1988c, Assessment of 
students with handicaps in vocational education: A 
curriculum-based approach ^ Arlington: American 
Vocational Association^ p. 17. 
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Trcin«r'tMaKjai 

CufTlculunrvBQsed Vocotlonol Assessment 



ABSTRACT 
Module 3 

Placing students in Vocational Edpcatibn Proimunm 



GOAL 



To assist local persoonel in implementing sys- 
tematic CBVA procedures determining ap- 
propriate vocational education placements for 
students with handicaps. 



CONTENT 
SUMMARY 



A series of steps is provided for a local CBVA 
team to Mow in helping a studtnt select the 
most appropriate program cqjtion m vocational 
education, A variety of assessment for place- 
ment methods is described and illustrated. 



KEYASSESSMEN'T 
QUESTIONS 



1., Is vocational education the best or most 
appropriate cumculum for the student? 

2. Which vocational education program is 
most appropriate Tor the student? 



Note , From L. Albright, 1988c, Assessment of 
students with handicaps in vocational education; A 
curriculum-based approach , Arlington; American 
Vocational Association, p. 19. 
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ABSTRACT 
Module 4 



187 



Planning the Student^s Vocational Education Program 



GOAL 



To increase participant knowledge and skills in 
conducting assessments wbicb identify the spe* 
ciai service needs of a studen^ in a vocational 
education program. 



CONTENT 
SUMMARY 



The module begins with a discussion of impor- 
tant differences between assessment for place* 
ment (Module 3) and assessment fbr program 
planning (Module 4) activities. This is foUowed 
with guidelines for conducting two types of as* 
sessment for program planning procedures: 

1. A content-specific survey; and 

2. A vocational program inventory. 

The results from these two assessments will ad* 
dress the key questions identified below. 



KEY ASSESSMENT 
QUESTIONS 



Is this particular student eligible to 
receive support services in his/her par* 
ticular vocational education program? 

In what particular areas of instructional 
remediation should support personnel 
focus to ensure success for a student in 
his/her vocational program? 

What is the nature and intensity of sup- 
port services necessary to deliver this in- 
structional remediation? 

What criteria should be applied to a 
student's performance which reasonably 
reflects the success norms of the vocation* 
al program? 



Note . From L. Albright, 1988c, Assessment of 
students with handicans in vocational education: A 



c urriculum-based approach , Arlington: 
Vocational Association, p. 21. 



American 



ERLC 



19S 



Trci>«r's Manual 

CurriCLlunrveosed Vocc3tlcxK3l Assessment 



ABSTRACT 
Module 5 



23 



188 



GOAL 



Monitoring Student Progress 

To assist local vocational education and special 
services personnel in establishing coordinated 
procedures for monitoring student !>erfor« 
mance in vocational education. 



CONTENT 
SUMMARY 



This module focuses on the use of assessment 
and monitoring procedures while the student is 
completing her/his vocational education 
program. Four basic considerations in estab- 
lishing a coordinated monitoring system are 
presented, along with example techniques and 
forms. 



KEY ASSESSMENT 
QUESTIONS 



1. Which areas of student performance need* 
to be monitored? 

2. What are the procedures to be used? 

3. How frequently should student progress 
be measured? 

4. Who is responsible for monitoring student 
progress? 



Note. From L. Albrignt^ 1988cy Assessment o f 
students wi 'i h handicaps in vocat i onal education; A 
curriculum-^ased approach ^ Arlington: American 
Vocational Association ^ p. 23. 
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ABSTRACT 
Module 6 

Planning Trangitional Services 

QOPiL To assist local vocational education and special 

services personnel in determining the special 
services needed by a student as she/he exits 
vocational education and enters new employ- 
ment and/or training environments. 



CONTENT 
SUiMMARY 



The module opens withal 'ef discussion of the 
rationale behind the transidonal services in* 
itiative for individuals with handicaps. The 
various options for transitional service delivery 
are subsequently described, as are the assess- 
ments which need to occur in planning and 
monitoring a student's transition. A case study 
of one student is presented to illustrate the 
types of assessment activities that take place 
during the tramitional period. 



KEY ASSESSMENT 
QUESTIONS 



1. In which outcomes (either postsecondary 
education, training or employment) will 
the student be functioning upon gradua- 
tion? 

2. What services will the student require at 
his/her selected option? 

3. Who will be responsible for monitoring the 
student as she/he transitions into the new 
environment? 

4. Now effective are the special services in 
helping the student succeed in the new 
environment? 



Note , From L. Albright, 1988c, Assessment of 
students with handicaps in vocational educati on: A 
curriculum-based approach , Arlington: American 
Vocational Association, p. 25 . 
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ABSTRACT 
Module 7 

Evaluating the Carriculum-Baged 
Vocational Assessment Process 



GOAL 



To assist local personnel in evaluating the ef- 
fectiveness of the CBVA process in determining 
appropriate vocational education and special 
services for students with handicaps. 



CONTENT 
SUMMARY 



A series of steps is provided for organizing, con- 
ducting and reporting an evaluation of a local- 
ly-based CBVA process. A team approach to 
this evaluation is advocated. Sample Instru- 
ments and forms are also pro'/ided for team 
member use. 



KEY ASSESSMENT 
QUESTIONS 



1. How effectively is the CBVA process work- 
ing in our district? 

2. What improvements are needed to 
strengthen the CBVA process? 



Note , From L. Albright, 1988c, Assessment of 
students with handicaps in vocational education; A 
curriculum-based approac h, Arlington: American 
Vocational Association, p. 27. 
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